12th  ICCRTS:  “Adapting  C2  to  the  21st  Century’ 


Paper  Title:  Hypothesis  Testing  of  Edge  Organizations: 
Empirically  Calibrating  an  Organizational  Model  for  Experimentation 

(Track  5:  Organizational  Issues) 

(ICCRTS  Paper:  1-092) 

(*Student  paper) 

Douglas  J.  MacKinnon* 

Stanford  University 

CIFE,  Terman  Engineering  Center,  MC:  4020 
(Room  5560) 

Stanford,  California 
94305-4020 

831.905.5913  (w)  /  650.723.4806  (f) 
dimackin@stanford.edu 

Marc  Ramsey 
Stanford  University 

CIFE,  Terman  Engineering  Center,  MC:  4020 
(Room  556H) 

Stanford,  California 
94305-4020 

650.723.8970  (w)  /  650.723.4806  (f) 
mramsey@stanford.edu 

Dr.  Raymond  E.  Levitt 
Stanford  University 

CIFE,  Terman  Engineering  Center,  MC:  4020 
(Room:  553M) 

Stanford,  California 
94305-4020 
650.723.2677  (w) 
ray.levitt@stanford.edu 

Dr.  Mark  E.  Nissen 
Naval  Postgraduate  School 
589  Dyer  Road,  Code  06/IS 
Monterey,  California 
93943-5000 
mnissen@nps.edu 


Acknowledgement:  This  research  is  sponsored  in  part  by  the  Office  of  the  Assistant  Secretary  of 
Defense  for  Networks  and  Information  Integration  (OASD/NII),  through  its  Command  &  Control 
Research  Program  (CCRP).  Several  complementary  parts  of  this  research  project  are  being 
coordinated  through  the  Center  for  Edge  Power  at  the  Naval  Postgraduate  School. 


1 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

2007 


2.  REPORT  TYPE 


4.  TITLE  AND  SUBTITLE 

Hypothesis  Testing  of  Edge  Organizations:  Empirically  Calibrating  an 
Organizational  Model  for  Experimentation 

6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Postgraduate  School, 589  Dyer  Road,  Code 
06/IS, Monterey, CA, 93943 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


3.  DATES  COVERED 

00-00-2007  to  00-00-2007 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Twelfth  International  Command  and  Control  Research  and  Technology  Symposium  (12th  ICCRTS),  19-21 
June  2007,  Newport,  RI 

14.  ABSTRACT 

This  paper  presents  our  ongoing  efforts  to  model,  simulate,  and  eventually  optimize  work  and  knowledge 
flows  in  Edge  organizations.  We  use  the  extended  POW-ER  3.2  framework  to  model  and  compare  two 
organizational  forms  (Edge  vs.  Hierarchy)  to  structure  participants  in  a  counter-intelligence  student 
exercise,  ELICIT  -  first  without,  and  then  with,  learning  micro-behaviors  enabled  in  POW-ER  3.2. 
Empirical,  experimental  data  on  learning  and  forgetting  from  observations  of  student  teams  conducting 
repeated  trials  of  the  AROUSAL  (Lansley,  1982)  business  simulation  exercise  at  Stanford  are  used  as  the 
basis  for  calibrating  agent  learning  and  forgetting  micro-behaviors  derived  from  the  cognitive  psychology 
literature.  We  then  compare  empirical  observations  of  student  teams  conducting  the  ELICIT  exercise  for 
both  Edge  and  Hierarchy  structural  configurations  with  outputs  from  POW-ER  3.2  computational 
simulation  models  representing  teams  executing  the  ELICIT  exercise  in  these  two  structural 
configurations.  This  initial  comparison  has  the  potential  to  further  calibrate  and  validate  POW-ER  for 
potential  use  in  analyzing  and  designing  C2  organizations.  Future  output  from  ELICIT  experiments  and 
other  empirical  data  on  learning  and  forgetting  will  augment  our  initial  comparison.  Calibrated  POW-ER 
3.2  learning  and  forgetting  micro-behaviors  will  improve  the  ability  of  POW-ER  to  model  and  simulate 
organization-level  C2  knowledge  flows  in  Edge  vs.  Hierarchical  organizations. 

15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF 

ABSTRACT 

OF  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

63 

Hypothesis  Testing  of  Edge  Organizations: 

Empirically  Calibrating  Organizational  Models  for  Experimentation 


Abstract 

This  paper  presents  our  ongoing  efforts  to  model,  simulate,  and  eventually  optimize  work  and 
knowledge  flows  in  Edge  organizations.  We  use  the  extended  POW-ER  3.2  framework  to  model 
and  compare  two  organizational  forms  (Edge  vs.  Hierarchy)  to  structure  participants  in  a  counter¬ 
intelligence  student  exercise,  ELICIT  —  first  without,  and  then  with,  learning  micro-behaviors 
enabled  in  POW-ER  3.2.  Empirical,  experimental  data  on  learning  and  forgetting  from 
observations  of  student  teams  conducting  repeated  trials  of  the  AROUSAL  (Lansley,  1 982) 
business  simulation  exercise  at  Stanford  are  used  as  the  basis  for  calibrating  agent  learning  and 
forgetting  micro-behaviors  derived  from  the  cognitive  psychology  literature.  We  then  compare 
empirical  observations  of  student  teams  conducting  the  ELICIT  exercise  for  both  Edge  and 
Hierarchy  structural  configurations  with  outputs  from  POW-ER  3.2  computational  simulation 
models  representing  teams  executing  the  ELICIT  exercise  in  these  two  structural  configurations. 
This  initial  comparison  has  the  potential  to  further  calibrate  and  validate  POW-ER  for  potential 
use  in  analyzing  and  designing  C2  organizations.  Future  output  from  ELICIT  experiments  and 
other  empirical  data  on  learning  and  forgetting  will  augment  our  initial  comparison.  Calibrated 
POW-ER  3.2  learning  and  forgetting  micro-behaviors  will  improve  the  ability  of  POW-ER  to  model 
and  simulate  organization-level  C2  knowledge  flows  in  Edge  vs.  Hierarchical  organizations. 


Introduction  and  Motivation 

Recent  rigorous  testing  and  comparing  of  Edge  (Alberts  and  Hayes,  2003)  and  Hierarchy  C2 
organizational  structures  has  improved  our  understanding  of  the  effects  of  Edge  versus  Hierarchy 
structures  in  terms  of  organizational  performance  (Orr  and  Nissen,  2006  and  Nissen,  2005). 
Earlier  efforts  focused  on  the  testing  and  analysis  of  the  efficacy  and  suitability  of  Edge  vs. 
Hierarchy  organizational  performance  under  Industrial  Age  vs.  21st  Century  conditions  (Orr  and 
Nissen,  2006)  yet,  did  not  address  the  topic  of  individual  learning  and  forgetting  and  its  impact  on 
organizational  performance. 

This  paper  continues  these  efforts  to  model,  simulate  and  eventually  optimize  work  and 
knowledge  flows  in  Edge  organizations,  by  comparing  empirical,  experimental  data  from  an 
exercise  (ELICIT),  for  both  structural  forms,  with  output  from  an  organization  simulation  model 
(POW-ER  3.2)  for  each  structural  form.  This  enables  us  to  further  calibrate  and  validate  the 
POW-ER  3.2  model  in  a  C2  context  to  examine  the  performance  differences  between  Edge  and 
Hierarchy  structures  engaged  in  similar  project-oriented  tasks.  We  will  leverage  the  recent 
extension  to  POW-ER  3.2  that  embeds  learning  and  forgetting  micro-behaviors  empirically 
determined  from  the  literature  and  calibrated  in  the  AROUSAL  exercise.  This  enables  us  to  move 
beyond  qualitative  reasoning  to  measure  and  report  the  quantitative  impacts  of  individual  learning 
and  forgetting  on  organizational  performance  outcomes  for  the  two  structures. 

We  begin  by  illustrating  our  concept  of  both  Edge  and  Hierarchy  organizations.  In  Power  to 
the  Edge,  Alberts  and  Hayes  (2003)  portray  an  agile  organizational  form  whose  high  level  of 
responsiveness  to  rapidly  changing  conditions  relies  on  decomposing  command  and  control  by 
moving  power  deliberately  to  the  “edge” — the  front  line  of  these  organizations  where  they  confront 
and  interact  with  their  environments.  We  therefore  envision  an  Edge  organization  to  resemble  a 
fully  connected  network  of  agents  who,  at  any  time,  gain  knowledge  from  any  other  agent  as 
shown  in  the  figure  below. 
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Figure  1:  Edge  Organization  showing  communication  links,  with  concomitant 
knowledge  flows,  between  all  agents. 


Edge  organizations  foster  shared  awareness  among  all  the  agents  as  well  as  open 
communication  of  knowledge  flow.  There  is  no  absolute  leader,  but  leadership  emerges,  at 
different  times,  based  on  a  meritocracy  among  capable  and  knowledgeable  players. 


Figure  2:  Hierarchy  Organization  showing  communication  links  and  concomitant 
knowledge  flows  between  agents  and  their  superiors. 


A  hierarchy  organization  enforces  communication  limits  between  players.  Hierarchies  are 
necessary  when:  highly  specialized  knowledge,  security,  or  permission  is  required.  This 
organization  can  be  less  agile  in  instances  where  each  agent  must  act  quickly  in  coordination  with 
the  actions  of  other  agents,  based  on  its  own  state  of  knowledge,  information  and  instructions, 
communicated  up  and  down  the  hierarchy,  respectively. 


3 


Efficient  knowledge  management  is  critical  to  mission  and  project  success  (Cole  1998,  Grant 
1996,  and  Spender,  1996),  yet  few  studies  have  explored  the  organizational  effect  of  individual 
skill  learning  together  with  individual  forgetting  as  a  project  continues  and  its  participants  improve 
their  skills  through  repeated  performance  of  tasks  or  lose  their  skills  through  non-performance. 

To  our  knowledge,  no  previous  research  has  explored  knowledge  flow  impacts  on  Edge 
organizations.  This  motivates  us  to  explore  and  compare  organizational  effects  of  individual 
learning  and  forgetting  in  both  Edge  and  Hierarchy  structures  as  skill  learning  and  forgetting 
occur  over  time  during  the  ELICIT  exercise,  and  apply  our  findings  to  further  calibrate  these 
effects  in  the  POW-ER  3.2  agent  microbehaviors. 

The  following  section  discusses  previous  efforts  to  model  and  compare  knowledge  flows 
between  Edge  and  Hierarchy  organizations. 


Background 

Individual  Knowledge  Flow 

Knowledge  flows  into  individuals  as  they  learn,  and  flows  out  as  they  forget;  yet  the 
psychological  and  neurological  processes  by  which  people  learn  and  forget  are  extremely 
complex.  We  would  be  naive  to  believe  that  we  might  explain  all  that  there  is  to  know  about  how 
humans  learn  and  forget  in  a  few  short  paragraphs.  We  will  narrow  our  concern  to  the  length  of 
time  required  for  an  individual  to  complete  a  skill,  or  processing  speed  (e.g.  Argote,  1999; 
DeKeyser,  1997;  and  Laird,  Newell,  and  Rosenbloom,  1987),  for  the  purpose  of  this  study. 
Conceptually,  we  do  not  attempt  to  quantify  knowledge  directly,  but  instead  measure  skill 
completion  time  as  a  surrogate  measure  for  knowledge  held. 


Figure  1:  Power  Law  of  Learning  Over  increasing  days  of  practice,  a  simple 
recognition  task  (time  required  to  recognize  sentences)  requires  ever  decreasing 
amounts  of  time  (as  found  in  Anderson,  (2005)). 


Skill  completion  -  or  task  completion  -  time  is  typically  modeled  in  terms  of  learning  curves 
(Ebbinghaus,  1913;  Anderson,  2005;  Sikstrom  and  Jaber,  2002;  and  Wixted,  2004).  Figure  1 
illustrates  the  Power  Law  of  Learning  (Pirolli  and  Anderson,  1985)  derived  from  empirical  studies 
that  appear  ubiquitously  in  cognitive  psychology  texts  (e.g.  (Anderson,  2005)). 

Time  delay  in  between  periods  spent  performing  a  task  causes  employees  to  forget.  The  rate 
at  which  forgetting  occurs  increases  with  task  complexity  and  with  simple  failure  to  recall  an  item 
or  procedure  with  some  frequency  (Jaber  and  Sikstrom,  2004).  As  with  learning,  forgetting  can 
be  modeled  by  a  function  based  on  a  power  law  (e.g.,  Wixted,  2004;  Wickelgren,  1 974;  and 
Ebbinghaus,  1913),  e.g.:  R(t)  =  at"b  where  t  is  time  and  a  and  b  are  scalars  (see  Figure  2). 
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Delay  in  days  (T) 

Figure  2:  Power  Law  of  Forgetting.  Over  time  what  is  known  decays  at  a 
negatively  accelerating  pace.  (Wickelgren,  1975;  Wixted  and  Ebbesen,  1991) 


Most  prior  research  on  forgetting  has  focused  on  relatively  simple  tasks,  such  as  list  learning, 
where  a  participant  is  asked  to  recollect  and  recite  memorized  items  from  a  list  and,  over  time, 
begins  to  forget  them  (e.g.,  Anderson,  2005).  Similar  effects  can  also  be  seen  in  the  recall  and 
performance  of  complex  skills  (Ericsson  and  Charness,  1994),  such  as  forgetting  in  a  practicing 
physician,  (Smith,  1978)  or  skill  decay  in  cardiopulmonary  resuscitation  (CPR);  (McKenna  and 
Glendon,  1985). 

We  will  maintain  our  focus  upon  the  learning  and  forgetting  of  a  complex  cognitive  skill  (such 
as  counterterrorism  analysis  rather  than  simplified  list  learning.  In  this  next  section,  we  further 
narrow  our  scope  toward  analyzing  skill  learning  and  forgetting  by  categorizing  different  skill  types 


Skill  Classification 

Not  all  skills  are  learned  by  individuals  with  equal  speed.  Dar-EI  et  al.  (1995)  classifies  skills  in 
the  four  following  categories:  (1)  highly  cognitive,  (2)  mostly  cognitive,  (3)  mostly  motor,  and  (4) 
highly  motor,  as  shown  below. 
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Figure  3:  Learning  Curves  for  Different  Types  of  Skill  (Source:  Dar-EI  et  al., 
1995).  As  skills  become  increasingly  cognitive  vs.  motor,  the  improvement  in 
performance  for  a  given  number  of  repetitions  is  more  pronounced  and  the  skill  level 
(the  inverse  of  the  time  to  takes  to  perform  the  task)  asymptotes  to  a  higher  multiple 
of  the  initial  skill  level. 
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Figure  3  illustrates  the  learning  rate  differences  among  four  classifications  of  skill.  We 
anticipate  that  this  classification  will  further  our  understanding  of  skill  learning  when  combined 
with  forgetting.  Later,  we  will  conduct  an  experiment  whose  empirical  data  will  be  plotted  against 
these  curves  to  attempt  to  classify  the  type  of  skill  being  conducted.  We  will  attempt  to  combine 
and  build  upon  these  theories  by  combining  learning  and  forgetting  with  skill  classification  to 
improve  our  ability  to  forecast  knowledge,  or  skill  level,  over  time.  Our  claims  of  theoretical  and 
practical  contribution  will  be  assessed  by  comparing  our  empirical  findings  from  the  AROUSAL 
exercise  to  the  Dar-EI  learning  curves  above. 

We  next  discuss  research  about  organization  level  skill  loss  and  gain  as  we  anticipate  that 
individual  learning  and  forgetting  rates  will  have  organization-level  effects. 

Organizations 

There  is  a  large  body  of  literature  on  information  flow  in  organizations,  going  back  to  the 
pioneering  work  of  Herbert  Simon  and  James  March  in  the  1950’s  (Simon  and  March,  1958). 
These  efforts  were  built  upon  though  increased  exploration  about  how  organizations  learned 
through  “encoding  inferences  from  history  into  routines  that  guide  behavior”  (Levitt  and  March, 
1988,  p.320),  by  “knowledge  that  becomes  embodied  in  the  organization’s  technology”  (Epple, 
Argote,  and  Devadas,  1991),  or  by  measuring  decreasing  costs  of  production  through  learning- 
by-doing  (Wright,  1936;  Goering,  1993;  and  Benkard,  2000).  However,  the  corresponding 
literature  on  knowledge  flow  in  organizations  is  still  emerging  (e.g.,  Levitt  et  al.,  1999,  Nonaka, 
1994,  and  Nissen,  2006)  and  pays  little  attention  to  the  impact  of  organizational  forgetting. 

Schreiber  and  Carley  (2004)  using  CONSTRUCT,  as  well  as  Reagans  et  al.  (2005),  have 
modeled  individual  knowledge  as  both  cognitive  and  transactive.  Transactive  knowledge  refers  to 
the  “meta-knowledge”  of  where  or  with  whom  different  kinds  of  knowledge  reside  (Wegner,  1987). 
They  seek  to  determine  the  impact  that  database  use  and  data  type  have  on  organizational 
performance  in  terms  of  levels  of  group  interactions  and  knowledge  sharing.  Our  research  has  a 
similar  goal,  but  will  take  a  different  approach  by  considering  the  deficits  of  forgetting.  We  seek  to 
improve  organizational  project  outcomes  by  accurately  predicting  the  impact  of  individual  learning 
and  forgetting  on  project  duration,  cost,  rework  volume  and  quality  risk. 

Researchers  have  also  analyzed  and  attempted  to  explain  individual  and  organizational 
levels  of  information  and  knowledge  flows  (Simon,  1950;  Argote,  1999;  and  Nissen,  2006). 
Knowledge,  seen  as  inflows  and  outflows  (Dierickx  and  Cool,  1989),  is  another  method  of 
analysis.  These  inflows  and  outflows  are  metaphorically  viewed  as  water  entering  and  exiting  a 
bathtub.  In  this  sense,  the  level  of  water  is  viewed  as  the  level  of  available  knowledge  to  the 
organization  and  the  amount  of  water  entering  and  exiting  the  bathtub  is  seen  as  knowledge 
improvement  and  knowledge  lost  respectively. 

We  consider  that  the  flow  of  an  organization’s  knowledge  can  be  modeled  through  learning 
and  forgetting  among  its  individual  participants.  And  when  those  individuals  frequently  perform 
their  skills,  there  is  no  loss  of  knowledge  and  potentially  some  growth.  When  skills  remain 
unused  or  dormant,  knowledge  slowly  erodes  over  time.  We  conceptualize  that  the  current  level 
of  knowledge  for  an  individual  is  demonstrated  though  the  required  time  for  the  individual  to 
accomplish  a  specific  skill-based  task.  We  consider  that  as  knowledge  level  improves,  a 
concomitant  increase  in  task  processing  speed  will  also  occur.  We  now  envision  a  method  to 
advance  from  qualitative  analysis  to  quantitative  analysis  of  organizational  knowledge  flow.  We 
will  analyze  the  organizational  effects  of  individual  learning  and  forgetting.  We  intend  to  use 
organization  simulation  (POW-ER  3.2)  to  demonstrate  the  impact  on  project  duration  and  process 
quality  as  dynamic  levels  of  individual  participants’  knowledge  level  occur. 

Our  research  questions  are: 

1 .  How  can  we  predict  the  effect  of  individual  learning  and  forgetting  on  organizational 
(or  group)  performance? 

2.  How  can  we  model  an  organization  whose  individuals  learn  and  forget? 
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Points  of  Departure 

Previous  attempts  to  model  and  simulate  Edge  vs.  Hierarchical  organizations  using  OrgCon 
(software  created  by  Burton  and  Obel,  1995)  and  SimVision  (Nissen,  2005;  Orr,  Nissen,  2006) 
were  conducted  to  determine  organizational  misfits  when  applying  Edge  vs.  Hierarchy 
organization  structures  to  given  tasks  and  contexts,  but  did  not  involve  agent  learning  or 
forgetting.  In  these  studies,  agents  were  assigned  fixed  skill  sets  and  each  organizational 
structure  was  imbued  with  different  simulation  parameters.  For  example,  Functional  Exception 
Probability  (FEP)  was  set  to  0.1,  as  well  as  high  centralization  and  formalization  with  low  matrix 
strength  for  the  Hierarchy  case;  and  FEP  was  set  to  0.2  with  low  centralization  and  formalization, 
and  high  matrix  strength,  for  the  Edge  case  (Nissen,  2005). 

It  was  determined  that  under  Industrial  Age  conditions,  Edge  organizations  performed  slightly 
better  than  their  Hierarchy  counterparts  for  project  length  (223  vs.  227  days)  and  cost  much  less 
(9B  vs.  12B  dollars),  yet  project  risk  was  much  higher  (.78  vs.  .36).  Under  21st  century  conditions, 
it  was  determined  that  for  project  duration  and  cost,  Edge  organizations  performed  much  better 
than  Hierarchies  (235  vs.  314  days)  and  (10B  vs.  16B)  while  relative  project  risks  remained 
unchanged  (.78  vs.  .36)  (Nissen,  2005p.  16). 

Our  research,  instead,  keeps  simulation  parameters  such  as  FEP,  matrix  strength  and 
centralization,  equal  across  the  two  organizational  forms,  in  order  to  isolate  any  differences  in 
organizational  performance  between  Edge  and  Hierarchy  forms  based  solely  on  structural 
differences  in  authority  relationships  and  information  flows.  In  addition,  we  incorporate  learning 
and  forgetting  micro-behaviors  for  participants  in  the  two  organizational  forms  to  assess  their 
impact  on  performance,  again  using  equal  parameterization. 

Learning  with  Forgetting 

Few  studies  have  been  published  illustrating  the  gains  and  losses  to  projects  as  their  employees 
learn  and  forget  skills  over  time,  or  as  they  turn  over  during  and  between  projects.  A  notable 
exception  is  (Ibrahim,  2005).  There  have  also  been  few  attempts  to  determine  how  to  manage 
this  knowledge  in  terms  of  interventions  such  as  mentoring,  training  or  OJT  (listed  from  smallest 
to  greatest  levels  of  available  research).  For  example,  Carley  and  Svoboda  (1996)  and  Carley 
(2001)  model  individual  learning  computationally,  and  model  organizations  that  can  adapt  (hire, 
fire,  redesign,  and  retask)  toward  an  increasingly  optimal  design  to  achieve  maximum 
organizational  performance.  Carley  also  models  the  reduced  impact  of  individual  knowledge  on 
performance  with  organizational  structural  changes. 

There  is  a  nascent  trend  in  the  literature  to  view  knowledge  as  a  supply  chain  or  knowledge 
chain  (Holsapple  and  Jones,  2004  and  2005).  This  line  of  research  to  date  is  qualitative,  using 
only  natural  language  descriptions.  Kim  (1993)  also  seeks  to  link  individual  learning  to 
organizational  learning  via  natural  and  metaphorical  thinking  and  examples.  However,  Kim’s 
efforts  do  not  extend  to  quantifying  the  effects  of  individual  learning  and  forgetting  on  project 
outcomes.  Our  contribution  will  be  to  extend  this  portion  of  the  literature  by  moving  beyond  the 
current  natural  language  descriptions  and  toward  a  more  quantitative  computational  perspective 
that  has  the  potential  to  predict  and  ultimately  manage  knowledge  inventory  optimally  in  project 
teams.  We  seek  to  improve  organizational  project  outcomes  by  determining  the  specific, 
quantitative  impacts  of  individual  learning  and  forgetting  on  organizational  performance. 

Ramsey  et  al.  (2006)  extended  POW-ER  to  add  learning  and  forgetting  capabilities.  During 
2005-06  we  obtained  theoretic  learning  and  forgetting  rates  from  Cognitive  Science  literature 
(MacKinnon  et  al.,  2006).  We  anticipate  that  each  agent’s  dynamic,  skill  completion  time  speed 
will  have  far-reaching  implications  throughout  the  organization  that  will  directly  affect  expected 
project  cost,  length,  rework  and  project  risk.  Thus,  we  wanted  to  calibrate  these  learning  and 
forgetting  rates  against  modern  organizational  knowledge-work  tasks  for  use  in  POW-ER.  Once 
these  calibrated  learning  and  forgetting  behaviors  rates  are  implemented  in  POW-ER,  we  can 
more  confidently  begin  to  quantify  how  dynamic,  individual  knowledge  will  affect  performance 
outcomes  at  the  organizational  level. 
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Current  Research 


Calibrating  Learning  with  Forgetting 

We  began  by  considering  how  we  might  measure  dynamic  knowledge  level  among  individuals.  It 
seems  that  one  clear  measure  of  knowledge  level  is  the  time  required  to  complete  a  complex 
task.  We  selected  a  business  case  simulation,  entitled  AROUSAL,  where  31  students  were 
asked  to  provide  quarterly  individual  functional  plans  as  well  as  integrated  company  business 
plans  by  choosing  from  among  an  array  of  possible  managerial  interventions.  The  four  functional 
roles  in  each  team  were:  marketing-sales,  operations,  human  resources,  and  finance.  Each 
participant  was  randomly  placed  in  a  four-person  group  and  given  one  of  these  roles  to  perform. 
(One  group  consisted  of  only  three  participants;  its  data  are  not  included  in  our  analysis.)  Each 
participant  was  directed  to  develop  his/her  individual  quarterly  functional  business  plan  and 
recommended  set  of  interventions.  Each  group  was  then  directed  to  convene  to  integrate  these 
plans  and  choose  coordinated  interventions  for  the  company.  Integration  was  non-trivial  because 
each  role  competed  for  limited  group  resources.  For  instance,  each  group’s  budget  for  each 
period  had  to  be  divided  among  investments  in  ongoing  operations,  marketing  expenses,  hiring 
new  employees  and  writing  proposals  (bids)  for  new  construction  jobs. 

The  simulation  ran  for  8  quarters,  during  which  we  recorded  the  time  taken  by  individuals  to 
develop  their  functional  plans  and  the  time  taken  by  groups  to  develop  their  integrated  plans.  The 
first  quarter  was  used  to  provide  the  players  with  training  in  how  to  input  interventions  and 
analyze  outputs,  so  we  have  not  included  data  from  this  quarter  in  our  analysis.  After  4  quarters, 
the  groups  were  asked  to  stop  playing  and  resume  some  time  later,  approximately  4  days,  at  their 
discretion.  This  delay  introduced  an  opportunity  for  us  to  measure  “forgetting”  of  previously 
acquired  skills  in  playing  each  role. 

Next,  we  show  how  our  data  on  individual  and  group  task  durations  over  time  compare  to  a 
set  of  four  learning  curves  for  different  kinds  of  skills:  highly  cognitive,  mostly  cognitive,  mostly 
motor,  and  highly  motor  (Dar-EI  et  al.,  1995).  Our  AROUSAL  data  were  normalized  against  the 
Dar-EI  data  based  on  the  time  for  the  first  iteration.  Normalized  predicted  durations  for  each 
period  from  our  “learning  and  forgetting-enabled”  POW-ER  3.2  simulation  model  of  the 
AROUSAL  exercise  are  plotted  on  the  same  graph. 
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Figure  3:  Dar-EI  et  al.  Learning  Curves  vs.  Normalized  AROUSAL  Individual  and 
Group  Learning  Rates  and  POW-ER  3.2  Predicted  Durations  Over  Multiple 
Trials.  These  data  show  the  effects  of  cognitive  learning,  the  effects  of  forgetting 
caused  by  a  delay  after  trial  Q4,  plus  the  re-acquisition  of  skill  following  the  delay. 
Note  the  fit  between  the  individual  and  group  AROUSAL  data  vs.  the  Dar-EI  “Hi  Cog” 
curve.  The  POW-ER  3.2  predicted  durations  for  individual  time  initially  fit  the  Dar-EI 
“Hi  Cog”  curve  well,  but  do  not  catch  up  with  the  recovery  in  empirical  AROUSAL 
durations  after  the  delay;  the  POW-ER  predictions  for  group  durations  exhibit  slower 
learning  rates,  comparable  to  Dar-EI’s  “More  Motor”  curve. 


Our  data  demonstrate  an  excellent  fit  to  Dar-EI’s  et  al.  (1995)  findings  for  “Highly  cognitive” 
skill  learning  as  shown  in  the  averaged  processing  times  for  both  individual  and  group  learning 
and  forgetting.  The  individuals  and  groups  each  exhibited  learning  behavior  in  their  skill 
completion  time  during  the  first  three  quarters.  (Recall,  quarter  1  data  is  not  included  because  it 
is  a  training  trial.)  Groups  and  individuals  each  showed  a  marked  increase  in  required  time  for 
the  quarter  5.  This  seems  due  to  the  3  to  4  day  delay  between  performing  the  first  three  and 
second  four  sets  of  simulation  runs.  The  increase  in  time  for  the  fifth  quarter  indicates  the  level  of 
skill  decay,  or  forgetting,  that  occurred  as  a  result  of  the  delay. 

We  note  that  our  empirical  data  plots  along  the  “cognitive  skill  type”  curve  for  both  the 
individual  and  the  group  data  for  the  second  and  third  quarters.  The  average  time  increase  for 
AROUSAL  groups  during  the  fourth  quarter  is  anomalous;  it  may  have  occurred  due  to  the 
groups’  growing  interest  in  performing  well  in  the  simulation  and  taking  more  time  to  integrate 
their  individual  plans.  Quarter  5  shows  an  increased  time  requirement  for  both  individuals  and 
groups,  with  the  individuals’  percentage  time  increase  being  significantly  higher  than  the  groups’. 
This  time  increase  follows  the  multi-day  break  taken  by  all  groups  between  sets  of  simulation 
runs.  Forgetting  seems  to  occur  less  drastically  among  the  groups  than  the  individuals  as  seen  in 
figure  3,  yet  both  the  individual  and  the  group  durations  return  quickly  to  approximate  their 
original  trend  curve  for  learning  of  cognitive  skills.  In  replicating  the  theoretic  findings  of  Dar-EI  et 
al.,  we  provide  compelling  evidence  to  validate  the  Dar-EI  “High  Cognitive”  learning  rates  that  we 
had  embedded  in  POW-ER  3.2  for  individual  learning  of  cognitive  skills. 

The  POW-ER  3.2  predicted  individual  durations  initially  fit  the  Dar-EI  “Hi  Cog”  curve  well,  but 
do  not  catch  up  with  the  recovery  in  empirical  AROUSAL  durations  after  the  delay.  The  relatively 
good  fit  between  the  individual  empirical  data  with  the  POW-ER  3.2  predicted  individual  learning 
and  forgetting  curve,  supports  our  claim  of  initial  calibration  and  validation  for  the  learning 
behaviors  in  the  POW-ER  3.2  model.  The  POW-ER  predictions  for  group  durations  exhibit  slower 
learning  rates,  comparable  to  Dar-EI’s  “More  Motor”  curve.  This  suggest  that  we  slightly  re¬ 
calibrate  POW-ER  for  more  rapid  learning  following  delay;  and  that  we  re-calibrate  POW-ER 
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micro-behaviors  to  try  to  get  group  rates  to  approximate  individual  rates  more  closely,  per  our 
empirical  findings. 

The  amount  of  skill  loss  caused  by  delay  between  runs  4  and  5  was  found  to  be  proportional 
to  the  length  of  the  delay.  For  every  day  of  delay  that  groups  waited  to  commence  the  final  4 
quarters,  the  time  required  for  the  groups  to  integrate  their  business  plans  grew  as  shown  below. 


Days  Delayed 


Figure  4:  Forgetting  Correlation  for  Aggregated  Group  Data  which  shows  the 
relatively  high  correlation  between  time  lost  in  completing  the  skill  after  a  given  delay 
in  time. 


From  the  regression  trend  line  in  Figure  4,  we  note  that  approximately  7.4  minutes  are  lost  for 
every  day  of  time  delay  between  tasks  (R2=0.7 1 33).  The  group  data  indicates  that  the  group  skill 
requires  39  minutes  on  average  to  accomplish.  From  this,  we  infer  that  for  every  day  in  delay, 
each  group  loses  approximately  19%  of  its  skill  in  terms  of  average  processing  speed  (7.4/39 
minutes).  (One  group  showed  skill  improvement  after  a  one-day  delay  which  resulted  in  negative 
time  lost.  Data  from  another  group  is  not  included  due  to  their  taking  two  breaks  in  delay  instead 
of  one.)  We  again  show  compelling  evidence  to  support  our  use  of  this  aggregated  finding  to 
calibrate  forgetting  of  cognitive  skills  in  our  POW-ER  model  because  of  this  high  correlation  found 
between  time  delay  and  the  fraction  of  skill  lost. 

We  next  modeled  the  AROUSAL  exercise  with  and  without  learning  by  the  POW-ER  3.2 
computational  agents.  The  AROUSAL  exercise  is  typically  conducted  in  about  90  minutes 
depending  on  the  team’s  ability.  The  smallest  “clock-tick”  in  POW-ER’s  discrete  event  simulation 
framework  is  currently  one  minute,  and  exception  handling  times,  waiting  time-outs  for 
“delegation  by  default”  decision  making,  and  other  simulation  parameters  were  originally 
developed  and  have  been  extensively  calibrated  for  tasks  with  durations  from  one  day  to  several 
days.  Ultimately  POW-ER  will  be  calibrated,  and  its  minimum  clock-tick  reset,  to  allow  for  tasks 
of  arbitrary  length.  In  the  meantime,  we  scaled  the  input  measures  up  from  minutes  to  days  and 
scaled  down  and  output  measures  back  to  minutes  in  making  the  comparisons  between 
simulated  vs.  empirical  data  for  the  two  organizational  forms. 

Table  1  compares  our  empirical  and  synthetic  experiment  findings. 
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Table  1.  AROUSAL:  Empirical  Data  vs.  POW-ER  Model  Output  Individual  and  group 
duration  data  shown  were  normalized  for  the  Empirical  data  and  the  POW-ER  model  in  the 
“without  learning”  case.  In  the  “learning-enabled”  case,  the  individual  empirical  and  the  POW-ER 
data  differ  by  1 .8%  and  the  group  data  differ  by  0.5%. 


Metric 

Individual  Data 

Group  Data 

Empirical 

POW-ER  3.2 

Empirical 

POW-ER  3.2 

Summed  individual  durations 
(based  on  initial  period, 
without  subsequent  learning) 

161  days 

161  days 

406  days 

406  days 

Duration  (with  learning) 

106  days 

103  days 

235  days 

233  days 

Percent  Savings  from 
Learning 

34.2% 

36.0% 

42.1% 

42.6% 

Table  1  compares  our  empirical  with  our  synthetic  output  at  both  the  individual  and  group 
levels.  Row  3,  columns  2  and  3,  contain  the  required  work  (161  days)  of  each  individual  both 
empirically  and  as  modeled  using  POW-ER  3.2.  Row  4,  of  columns  2  and  3,  illustrate  the 
reduced  duration  of  the  work  as  a  result  of  learning  by  the  individuals.  Row  5  contains  the 
percent  savings,  both  observed  and  output  from  the  model,  that  result  from  learning.  Columns  4 
and  5  include  the  same  comparisons  taken  at  the  group  level. 

The  161  days  shown  for  the  duration  for  the  empirical  data  was  determined  by  multiplying  the 
original  average  exercise  time  required  (23  days),  by  the  number  of  quarters  to  be  played  —  in 
this  case  seven  —  i.e.,  (23  *  7).  The  duration  with  learning,  shown  in  the  next  row,  was 
developed  by  observing  student  teams  who  performed  each  subsequent  quarter  requiring  less 
time  than  the  previous  quarter,  with  the  exception  of  the  quarter  that  followed  the  production 
break  between  Q4  and  Q5.  The  average  total  time  required  by  the  individuals  was  106  days,  or  a 
34.2%  savings  resulting  from  learning.  Groups  required  58  days  each  quarter,  for  a  total  of  406 
days  —  (58  *  7).  Their  duration  from  learning  also  caused  a  decrease  which  totaled  233  days. 

The  POW-ER  model  began  with  the  same  required  time  yet  with  the  learning  and  forgetting 
micro-behavior  embedded,  forecasted  a  savings  of  36.0%  among  the  individuals  and  42.6  % 
among  the  groups  over  the  seven  quarters.  The  difference  between  the  empirical  data  and  the 
model  output  in  the  learning  case  may  be  accounted  for  by  differences  among  teams  of 
individuals. 

The  following  section  illustrates  POW-ER  3.2  predictions  for  individual  quarterly  durations 
followed  by  group  durations  in  the  AROUSAL  exercise. 
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Figure  5:  POW-ER  3.2  Predicted  Individual  Durations  for  Each  Quarter  which 
shows  very  high  correlation  between  the  empirical  and  synthetic  output. 
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Figure  6:  POW-ER  3.2  Predicted  Group  Durations  for  Each  Quarter  which  also 
shows  relatively  high  correlation  between  the  empirical  and  synthetic  output. 


The  group  durations  fit  the  relative  trend  of  the  empirical  data.  We  attribute  the  difference  to 
relative  differences  among  teams.  From  these  results,  we  claim  that  the  learning  and  forgetting  micro¬ 
behaviors  in  POW-ER  are  qualitatively  validated  and  tentatively  calibrated  to  reasonable  rates  of 
individual  learning  and  forgetting  for  cognitive  tasks  at  both  the  individual  and  group  levels. 

We  next  discuss  our  methodology  to  further  refine  our  calibration  and  validation  of  POW-ER 
3.2  by  using  more  empirical  data  obtained  through  another  exercise  entitled  “ELICIT.”  This  effort 
serves  our  purpose  of  refined  calibration  and  validation  but  also,  offers  the  opportunity  to 
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compare  how  two  different  organizational  forms  differ  in  their  performance  when  their  individuals 
learn  and  forget.  We  will  provide  an  overview  of  ELICIT,  followed  by  a  description  of  our 
modeling  efforts. 


POW-ER  Model:  ELICIT 

Exercise  Overview 

The  ELICIT  exercise  was  created  for  experimentation  in  command  and  control  settings  to 
compare  alternative  organizational  forms  (e.g.,  Hierarchy  vs.  Edge).  ELICIT  simulates  an  anti¬ 
terrorist  organization  that  must  collectively  analyze  a  series  of  “factoids”  — simple  statements 
related  to  an  impending  terrorist  attack  obtained  via  (unspecified)  counter-terrorism  intelligence 
sources — distributed  to  members  of  the  organization  in  order  to  specify  the  impending  terrorist 
attacks  precisely  in  terms  of:  who,  what,  when,  and  where.  We  simulate  ELICIT  initially  without 
individual  learning  and  forgetting  using  data  from  three  trials  conducted  in  June  2006.  We  then 
embed  agent-based  learning  and  forgetting  algorithms  (calibrated  from  our  AROUSAL  exercise) 
in  the  agents  in  our  ELICIT  POW-ER  model  to  predict  and  compare  the  differences  in 
organizational  performance  between  Edge  and  Hierarchy  organizations,  with  and  without 
individual  learning. 

ELICIT  is  played  using  a  set  of  instructions  for  either  the  Hierarchy  or  Edge  organizational 
form  (Parity  Communications,  2006).  Players  in  the  Hierarchy  form,  for  example,  are  instructed 
that  they  have  each  been  assigned  to  a  specific  team  of  four  players  each  of  whom  is  to  resolve  a 
specific  part  of  the  terrorist  attack  plot.  The  team  names  are:  Who,  What,  When,  Where.  Each  of 
these  teams  has  a  leader  assigned  who  coordinates  information  with  the  overall  Coordinator. 

The  Edge  players  are  not  assigned  to  specific  teams,  nor  is  there  an  overall  Coordinator. 

Each  member  of  both  the  Hierarchy  (H)  and  Edge  (E)  teams  receives  four  factoids  in  total  to 
assist  in  their  identification  of  the  terrorist  threat.  They  are  distributed  two  to  every  player  initially, 
followed  by  two  additional  distributions  (or  waves)  of  one  factoid  each  time.  The  initial  wave  of 
factoids  commences  shortly  after  the  game  commences  and  occurs  again  at  five-minute  intervals 
(at  time:  0,  5,  and  10  minutes  respectively).  There  are  differences  in  how  these  factoids  can  be 
communicated  among  the  players.  Hierarchy  players  may  only  post  to  and  view  from  their  own 
team’s  website,  whereas  Edge  players  can  post  to  and  view  from  any  team’s  website.  Any  player 
in  either  organizational  form  may  however,  share  their  factoids  with  any  other  player.  In  the 
Hierarchy  form,  the  Coordinator  is  the  only  player  who  may  view  all  the  team’s  websites.  The 
Coordinator  may  then  decide  to  share  particular  and  appropriate  factoids  with  a  specific 
Hierarchical  team.  Each  team  is  comprised  of  17  players  who  may  share  their  knowledge  using 
only  electronic  methods  via  various  electronic  channels  as  explained  above.  We  therefore 
hypothesize  that  a  bottleneck  of  information  flow  can  occur  as  a  result  of  requiring  the  Hierarchy 
Coordinator  to  view  and  then  share  other  teams’  factoids. 

Each  player  is  then  given  up  to  60  minutes  to  identify  the  terrorist  threat  correctly  by 
combining  multiple  and  corroborating  factoids  that  seem  to  convey  the  most  plausible  and 
logically  correct  scenario.  Each  player  may  identify  all  or  part  of  the  terrorist  threat  at  any  time 
during  the  exercise.  The  exercise  concludes  when  all  players  have  correctly  identified  the 
terrorist  threat. 


Modeling  Details:  Hierarchy  vs.  Edge 

We  begin  with  a  clean  slate  approach,  taking  into  consideration  the  agents  and  the  tasks  they 
must  complete.  Agents  of  similar  abilities  and  with  similar  task  assignments  were  modeled  in 
POW-ER  3.2.  Multiple  agents  may  be  grouped  together  if  assigned  to  accomplish  the  same  task 
or  tasks.  Dissimilar  agents  of  different  skill  levels  must  be  assigned  separately  to  tasks.  We  next 
create  tasks  for  the  agents,  or  groups  of  agents,  such  as  coordinating  or  analyzing.  The  Overall 
Coordinator  in  the  Hierarchy  organization  is  assigned  the  task  of  “overall  coordination,”  the  team 
leaders  “coordinate”  their  individual  team,  and  team  members  “analyze”  their  factoids. 
Assignment  connection  arcs  are  used  to  identify  task  responsibility.  Each  successive  task  is 
connected  in  series  until  the  simulation  is  finished.  “Communication  links”  are  added  to  show 
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who  has  the  ability  to  communicate  with  whom.  “Knowledge  links”  are  added  to  indicate  who  has 
access  to  partially  or  fully  shared  knowledge  repositories  such  as  synthetic  websites.  This  style 
of  modeling  follows  a  reasonable  fit  to  the  ELICIT  methodology  of  the  exercise.  ELICIT  tasks  are 
discretely  assigned  to  each  individual  or  team  and  are  continually  worked  upon  until  each 
assigned  task  is  accomplished.  The  C2  processes  of  organizational  knowledge  sharing  are  also 
effectively  reproduced  using  POW-ER  3.2  links. 

In  the  Edge  form  there  is  no  assigned  coordinator  and  so  one  player  was  added  to  “Team  A” 
in  our  model  to  ensure  the  models  remained  equal  in  terms  of  available  Full-Time-Equivalent 
staffing.  We  also  ensured  that  each  overall  team  was  assigned  an  equal  amount  of  work.  The 
POW-ER  models  of  the  Hierarchy  and  Edge  organizations  used  in  our  POW-ER  3.2  model  of  the 
ELICIT  exercise  are  illustrated  in  figures  2  and  3  below. 

All  agents  for  each  organizational  structure  were  given  the  skill  level  of  “low.”  This  level  can 
be  allowed  to  increase  dynamically  as  the  skill  is  used  to  perform  tasks  or  can  decrease,  as  the 
skill  remains  dormant.  It  can  also  be  kept  fixed  for  the  duration  of  the  simulation  by  disabling 
learning  and  forgetting  in  our  model.  An  initial  skill  designation  of  “low”  provided  us  with  a 
noticeable  difference  between  successive  runs  of  the  same  work  processes  when  agent’s  skill 
either  increased  or  decreased. 


Overall  Coordination 


Analyze  Who 


Coordinate  What 


Analyze  What 


Coordinate  When 1 


Analyze  When 


'Coordina  e  Where 


Analyze  Where 


jj _  ^r* 
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Figure  7:  Hierarchy  Organization  of  the  ELICIT  game  showing  communication 
links  between  the  task  boxes  as  well  as  knowledge  links  between  all  agents  and  the 
ELICIT  websites.  An  overall  coordinator  with  a  two-level  organization  is  also 
modeled. 


ELICIT  game  players  can  “pull”  knowledge  from  synthetic  websites  to  refine  their 
identification  of  the  threat  scenario  as  discussed  above.  We  envision  an  opportunity  to  take 
advantage  of  a  unique  capability  in  POW-ER,  that  does  not  exist  in  SimVision,  by  modeling 
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knowledge  networks  using  knowledge  links  (shown  between  agents  and  websites)  that  allow  a 
player’s  (agent’s)  functional  exceptions  to  be  handled  by  the  highest  skilled  agent  (a  website  in 
this  case)  to  which  that  player  is  linked.  The  use  of  knowledge  networks  allows  agents  to  seek 
out  information  from  agents  (including  online  knowledge  bases  or  web  sites)  of  higher 
skill/knowledge,  when  such  resources  are  available.  This  method  of  meta-knowledge  or 
knowledge  networks,  although  not  an  exact  fit  for  the  kinds  of  knowledge  processing  that  occur  in 
an  ELICIT  game,  seems  a  reasonable  approximation.  This  methodology  represents  a  reasonable 
approximation  for  a  possible  future  version  of  ELICIT  if  the  game  were  to  allow  for  a  “system  of 
reputation”  or  source  verification  to  be  used  by  future  players. 


±1  jf* 

^Hierarchy^dg^^ 

Figure  8:  Edge  Organization  of  the  ELICIT  game  showing  direct  communication 
between  all  tasks  (boxes)  as  well  as  knowledge  links  between  agents  and  synthetic 
websites,  within  a  flat,  “Edge”  organization. 


In  the  Edge  case,  all  the  players  were  assigned  the  “team  member”  role  as  a  means  of 
modeling  equal  decision  making  responsibilities,  whereas  in  the  Hierarchy  case,  the  Overall 
Coordinator  is  assigned  the  “Program  Manager”  role.  The  team  leaders  are  assigned  “Leader” 
roles  and  the  remaining  agents  are  assigned  team  member  roles  to  account  for  their  decreased 
decision  making  responsibilities.  As  in  the  AROUSAL  model,  we  scaled  our  assigned  work  within 
the  POW-ER  model  in  days  although  the  ELICIT  game  is  typically  played  for  about  60  minutes 
depending  on  the  protocol  involved.  We  again  directly  translate  inputs  and  outputs  back  and  forth 
from  minutes  to  days  and  back  to  minutes  for  analysis. 

POW-ER  also  has  the  ability  for  the  user  to  invoke  and  control  certain  simulation  parameters 
such  as:  communication  probability,  noise  probability,  functional  error  probability,  and  project 
exception  probability  and  many  others.  We  set  the  communication  probability  to  0.6  for  both  the 
Hierarchy  and  the  Edge  forms  within  POW-ER  3.2  to  account  for  a  relatively  high  level  of  required 
communication  and  because  we  wish  to  determine  the  difference  resulting  the  two  organizational 
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structure  forms  without  increasing  (or  decreasing)  other  model  parameters  for  either 
organizational  form.  We  set  the  functional  exception  probability  in  each  case  to  0.5  to  represent 
relatively  high  amounts  of  knowledge  processing  activity  within  each  organizational  form.  All 
other  parameters  remained  at  their  default  values  and  were  equal  between  the  two  organizational 
forms. 


Results 

ELICIT  Empirical  Data 

The  fully  instrumented  data  output  which  recorded  each  player’s  actions,  enabled  our  analysis  of 
the  frequency  of  knowledge  ( factoid )  sharing  via  synthetic  websites,  and  the  occurrence  of  correct 
responses  regarding  target  identification  in  terms  of  who,  what,  when,  and  where.  The  following 
graph  illustrates  the  quantitative  difference  between  Hierarchy  and  Edge  organizational  forms 
though  the  comparison  of  students’  accurate  answers  given  during  each  ten-minute  interval  of 
each  game.  Data  from  three  rounds  of  the  ELICIT  game  are  available  at  present.  Two  of  these 
rounds  were  conducted  using  a  Hierarchy  organization.  The  hierarchy  data  report  the  average 
of  the  two  ELICIT  rounds  played  in  the  “hierarchy”  structure.  The  other  round  implemented  an 
Edge  organization. 


Correct  Responses  in  10-minute  Intervals 

For  Different  Organizational  Forms 


10  20  30  40  50  60 

10-Minute  Intervals  (non-cumulative) 


Figure  9:  Hierarchy  vs.  Edge  Organization  empirical  output  from  the  ELICIT 
game  showing  the  total  number  of  correct  answers  given  by  all  members  of 
Hierarchy  or  Edge  organizations  during  successive  ten  minute  intervals. 


The  numeric  data,  included  in  Appendix  A,  also  indicate  that  the  Hierarchical  organizations 
never  produced  a  completely  correct  response  to  the  terrorist  threat.  The  Edge  organizations 
produced  completely  correct  responses  at  minutes:  16,  17,  18,  19,  20,  21,  32,  and  49  after  the 
exercise  was  begun.  The  empirical  data  illustrated  above  reveal  a  substantial  difference  between 
the  two  organizational  forms  in  terms  of  the  frequency  of  agent’s  correct  answers  produced  over 
time. 

The  small  number  of  games  played  limits  any  claim  of  statistical  significance.  We  will 
therefore  search  for  reasonable  conclusions  about  the  two  organizational  forms  that  arise  from 
the  empirical  findings.  These  initial  conclusions  will  guide  our  assessment  and  initial  veridicality 
of  our  model  output.  We  anticipate  that  future  ELICIT  rounds  will  provide  us  data  for  more 
rigorous  statistical  testing  and  increase  the  reliability  of  our  conclusions. 
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This  next  section  uses  the  evidence  above  to  qualitatively  compare  POW-ER  3.2  model 
output.  We  use  the  learning  and  forgetting  algorithms  developed  from  AROUSAL  to  model  the 
ELICIT  Edge  and  Hierarchy  organizations.  We  analyze  the  model  output  to  test  whether  we  can 
begin  to  validate  the  learning  behaviors  in  POW-ER  qualitatively  using  the  empirical  ELICIT  data, 
and  show  that  Edge  organizational  performance  may  surpass  Hierarchical  performance  as  a 
result  of  individual  learning. 


POW-ER  Model  Predictions 

Table  2  below  provides  a  comparison  of  average  POW-ER  model  predictions  from  1,000 
simulation  runs  based  on  the  ELICIT  exercise,  using  the  Hierarchy  vs.  Edge  organizational  forms, 
assuming  no  individual  learning  occurs  during  the  exercise.  We  conducted  these  runs  to  set  a 
reasonable  baseline  of  organizational  performance  without  invoking  our  learning  and  forgetting 
algorithm. 

Table  2.  POW-ER  Output:  Hierarchy  vs.  Edge  Performance  (without  learning)  Simulated 
duration  is  qualitatively  comparable  to  empirical  results  in  terms  of  the  timing  and  frequency  of 
correct  answers  among  the  two  organization  forms.  Computation  of  duration  and  work  effort  was 
conducted  in  days,  which  are  translated  to  minutes  in  the  student  game  as  explained  above.  The 
average  of  1000  runs  is  listed  first  with  standard  deviations  in  parentheses.  Metrics  that  are 
statistically  significantly  different  for  the  two  forms  are  marked  with  an  asterisk  and  the  result  of 
the  “higher  performing”  structure  in  each  row  is  shown  in  bold  font. 


Metric 

Hierarchy 

Mean  (Std.  deviation) 

Edge 

Mean  (Std.  deviation) 

Duration* 

96.4  (8.0)  days 

90.6  (10.1)  days 

Coordination* 

405.0  (13.2)  days 

289.6  (11.9)  days 

Rework 

276.7  (19.6)  days 

274.9  (25.5)  days 

Functional  Exception  Work 

1373  (n/a)  days 

1378  (n/a)  days 

Total  Work* 

2280.3  (29.6)  days 

2170.9  (32.5)  days 

Functional  Risk 

.798  (.012) 

.801  (.015) 

Process  Quality  Risk 

.475  (.004) 

.475  (.005) 

Cost* 

1802.0  (23.5)  K 

1737.1  (26.0)  K 

In  the  three  student  team  runs  of  the  ELICIT  exercise,  play  was  terminated  after  60  minutes 
without  either  team  having  “completed”  the  exercise.  That  is,  neither  team  converged  to  a  correct 
identification  of  ail  four  parameters  of  the  anticipated  terrorist  plot  that  was  to  be  interdicted, 
although  some  players  had  made  fully  or  partially  correct  identifications  at  various  points  during 
the  exercise.  Thus,  any  attempts  to  claim  calibration  in  terms  of  even  “qualitative  consistency” 
are  quite  tentative  at  present. 

The  POW-ER  predictions  from  these  initial  prototype  models  and  our  first  three  sets  of 
empirical  ELICIT  data  show  plausible  qualitative  consistency  with  the  empirical  data  for  duration, 
compared  with  the  three  rounds  of  the  ELICIT  game.  For  instance,  it  seems  likely  that  from  the 
empirical  study  that  the  Hierarchy  game  would  have  required  much  more  time  for  the  players  to 
have  correctly  identified  the  plot,  whereas  seven  of  the  student  Edge  players  had  already 
correctly  identified  all  parameters  of  the  terrorist  plot  at  the  sixty  minute  point  (see  Appendix  A). 
The  other  POW-ER  output  metrics  listed  also  appear  to  indicate  qualitatively  correct  differences 
between  the  outcome  metrics  for  Hierarchy  and  Edge  organizational  forms,  based  on  the 
theoretically  claimed  advantages  of  Edge  Organizations  as  well  as  prior  computational  modeling 
experiments  described  in  the  first  section  (Orr  and  Nissen,  2006  and  Nissen,  2005). 
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ELICIT  Model  with  Learning  and  Forgetting 

We  next  took  the  same  POW-ER  models  and  sequentially  executed  them  three  times  in 
succession  using  no  gap  between  rounds  one  and  two,  and  a  five  day  gap  between  rounds  two 
and  three  to  allow  for  learning  and  forgetting  to  occur.  Our  simulation  findings  are  shown  below. 


Table  3.  POW-ER  Output:  Hierarchy  vs.  Edge  with  and  without  Learning  and  Forgetting. 

The  “No  Learning”  baseline  results  from  Table  2  are  multiplied  by  three  to  compare  with  the  three 
trials  that  have  learning  and  forgetting  enabled.  All  “no  learning”  results  are  the  mean  of  100  runs 
shown  with  the  standard  deviation  in  parentheses.  Metrics  that  exhibit  statistically  significant 
differences  between  the  “With  learning  and  forgetting”  cases  for  Hierarchy  vs.  Edge  are  denoted 
by  an  asterisk. 


Metric 

Hierarchy  (3  Rounds) 

Mean  (Std.  deviation) 

Edge  (3  Rounds) 

Mean  (Std.  deviation) 

No  learning 

With  learning 
and  forgetting 

No  learning 

With  learning 
and  forgetting 

Duration  (days)* 

289.2  (8.0) 

244.9  (14.9) 

271.8  (10.1) 

210.0  (20.5) 

Coordination  (days)* 

405.0  (13.2) 

454.9  (15.4) 

289.6  (11.9) 

313.0  (11.8) 

Rework  (days) 

276.7  (19.6) 

276.8  (20.4) 

274.9  (25.5) 

274.2  (27.9) 

Functional  Exception 
Work  (days) 

1373  (n/a) 

1378.3  (n/a) 

1378  (n/a) 

1373.5  (n/a) 

Total  Work  (days)* 

2280.3  (29.6) 

2338.2  (31.8) 

2170.9  (32.5) 

2193.7  (36.0) 

Functional  Risk 

.798  (.012) 

.799  (.012) 

.801  (.015) 

.801  (.016) 

Process  Guality  Risk 

.475  (.004) 

.474  (.004) 

.475  (.005) 

.475  (.005) 

Cost  ($K)* 

1802.0  (23.5) 

1297.7(46.3) 

1737.1  (26.0) 

1191.9  (53.1) 

The  POW-ER  3.2  ELICIT  model  provides  many  kinds  of  outputs  as  illustrated  above.  We  will 
continue  to  focus  on  project  duration  to  maintain  consistency  and  verifiability.  The  model  output 
indicates  that  the  expected  duration  of  the  ELICIT  game  for  three  rounds  (two  rounds  in 
succession  followed  by  a  break  of  five  days  followed  by  the  third  round)  is  highest  for  the 
Hierarchy  organization  when  learning  is  not  invoked.  The  Hierarchy  organization,  with  learning 
and  forgetting  invoked,  demonstrates  a  statistically  significant  decrease  in  required  duration.  The 
Edge  organization  also  demonstrates  a  statistically  significant  decrease  in  duration  when  learning 
and  forgetting  is  invoked.  We  find  also  that  the  Edge  organization  performs  better  than  the 
Hierarchy  organization  when  compared  in  the  simulation  runs  with  (or  without)  learning.  This 
matches  the  empirical  ELICIT  data  for  these  three  runs  of  the  exercise.  This  is  in  line  with  our 
finding  that  Edge  organizations  are  predicted  to  perform  better  than  Hierarchical  organizations 
when  learning  is  present  —  and  Edge  organizations  learn  more  rapidly! 

For  external  validity,  we  compare  expected  ELICIT  game  lengths  by  dividing  the  duration  by 
three.  This  reveals  average  game  lengths  of  approximately  96.4  and  81 .6  minutes  in  the  case  of 
the  ELICIT  Hierarchy  organization  without  and  with  learning  respectively.  This  is  a  savings  of 
approximately  15.4%.  Edge  game  lengths  are  calculated  to  be  90.6  and  70.0  minutes  for  the 
Edge  organization  without  and  with  learning  respectively.  This  is  a  savings  of  approximately 
22.7%.  This  suggests  that  when  individuals  learn  in  both  structures,  an  Edge  organization  can 
improve  its  performance  through  reducing  its  required  project  duration  by  an  additional  7.3%  over 
a  Hierarchical  organization. 
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Figure  9  above  supports  the  initial  face  validity  of  the  POW-ER  simulation  model  with  learning 
and  forgetting  when  applied  to  model  this  command  and  control  synthetic  task  in  two  structural 
configurations.  We  will  continue  to  validate  and  calibrate  the  ELICIT  model  as  more  empirical 
data  becomes  available  and  more  is  learned  about  the  modeled  organization  differences. 


Discussion 

POW-ER  is  intended  to  model  communications  and  exceptions  for  varying  organizational  forms. 
At  present,  the  only  forms  of  communication  used  in  ELICIT  are  selective  posts  and  shares,  and 
there  is  no  feedback  to  players  on  the  correctness  or  otherwise  of  their  “identification”  assertions. 
The  players  do  not  receive  nor  respond  to  exceptions  in  the  traditional  sense  of  asking  directly  for 
assistance  from  a  manager  or  knowledgeable  peer,  so  we  need  to  approximate  this  behavior 
through  their  communications  to  one  another  and  their  postings  and  reading  of  data  from  a  game 
website.  Moreover,  as  described  above,  neither  the  game  nor  the  protocol  for  running  this 
version  of  it  provides  a  stopping  point  for  teams  that  have  already  achieved  correct  answers. 

This  lack  of  agent  feedback  directly  affects  the  outcome  of  the  game.  For  example,  looking 
at  the  available  data  for  the  Edge  scenario  (Appendix  A),  two  players  settled  upon  the  correct 
answer  early.  Five  other  players  also  offered  completely  correct  answers  early  in  the  game,  yet 
rendered  incorrect  responses  afterwards.  The  remaining  ten  players  were  correct  on  some  of  the 
answers  at  various  points  of  the  game  yet  failed  to  converge  on  the  correct  target  identification. 

Thus  far  it  has  been  shown  that  humans  perform  better  on  the  ELICIT  game  in  the  Edge 
structure  as  measured  by  our  "number  of  correct  answers  per  1 0  minute  interval"  metric. 

This  study  remains  challenging  as  we  attempt  to  leverage  our  ELICIT  modeling  efforts  (with 
learning)  based  on  limited  empirical  results.  We  will  continue  to  refine  our  modeling  and 
simulation  of  the  ELICIT  game  in  POW-ER  3.2  for  both  types  of  organizations,  and  look  for 
opportunities  to  execute  more  empirical  runs  of  ELICIT.  This  will  provide  further  comparisons 
between  the  two  organizational  forms  and  further  calibration  and  validation  for  POW-ER. 

At  this  time  we  claim  only  face  validity  for  comparability  between  the  POW-ER  model  and  the 
ELICIT  exercise  it  is  attempting  to  emulate.  We  claim  to  have  obtained  plausible  qualitative 
agreement  of  model  predictions  for  Edge  vs.  Hierarchy  with  learning  enabled  from  one 
experiment,  given  the  current  implementations  and  limitations  of  both  ELICIT  and  POW-ER. 
Ongoing  comparisons  will  be  made  and  analyzed  between  empirical  ELICIT  output  and  POW-ER 
simulation  output  for  both  a  single  trial  (no  learning)  and  multiple  trials  (with  learning)  of  the 
ELICIT  exercise.  We  will  also  conduct  additional  empirical  validation  experiments  in  which  we 
model  and  simulate  other  work  tasks  to  validate  POW-ER  with  learning  and  forgetting  agents. 
This  will  support  ongoing  calibration  of  the  workflow  model  in  POW-ER  3.2  and  of  the  learning 
micro-behaviors  that  have  recently  been  embedded  in  POW-ER.  Analysis  of  our  ongoing 
research  will  be  reported  at  the  conference. 


Conclusions 

This  paper  reports  on  our  continuing  efforts  to  understand  the  performance  effects  of  Edge 
versus  Hierarchy  structural  forms  through  cross-calibrated  empirical  micro-experiments  and 
computational  modeling  experiments.  This  set  of  cross-validation  experiments  employs  synthetic 
group  experiments  in  two  small  group  exercises  and  organizational  simulations  of  Edge  vs. 
Hierarchical  forms  in  these  two  games  with  and  without  learning  by  agents,  to  cross-validate, 
calibrate  and  refine  POW-ER  parameters.  We  described  our  continuing  steps  in  specifying  the 
key  variables  that  effect  work  flow,  knowledge  flow  and  organizational  learning  in  both  Edge  and 
hierarchy  organizations.  The  dynamics  of  individual  knowledge  gained  and  lost  in  organizations 
are  captured  and  are  thus  able  to  extend  our  understanding  of  organizational  learning  through 
this  extension  to  the  POW-ER  model  framework. 

These  experiments  provide  new  evidence  for  some  of  the  predicted  performance  differences 
between  Edge  and  Hierarchy  C2  organizations  both  empirically  and  synthetically,  and  contribute 
toward  an  improved  knowledge  of  performance  effects  for  Power  to  the  Edge  02  organization 
structures. 
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(Appendix  B  contains  proposed  changes  to  the  ELICIT  exercise  that  may  enhance  its 
veridicality  as  a  means  to  test  the  effects  of  alternative  organizational  structures  on  team 
performance  for  C2  tasks,  and  to  enrich  our  understanding  of  the  impacts  of  participant’s  learning 
and  forgetting  rates.) 


Future  Steps 

We  intend  to  validate  and  calibrate  POW-ER  further  as  more  ELICIT  data  become  available,  so 
that,  through  POW-ER,  we  may  generate,  model,  and  test  novel  hypotheses  about  Edge  and 
other  alternate  organizational  forms.  We  will  also  begin  to  develop  representation  and  reasoning 
to  model  and  simulate  knowledge  interventions  such  as  training  and  mentoring  to  further  explore 
the  effects  of  such  organizational  knowledge  flow  investments. 
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Appendix  A  -  ELICIT  game  data 

Data  output  from  three  previous  student  rounds  of  the  ELICIT  Exercise  (17,  22,  and  23  June 
2006)  are  provided  in  the  following  three  appendices.  Correct  responses  in  a  single  category  are 
indicated  by  purple  shading.  Bright  green  shading  indicates  responses  that  are  correct  in  every 
category. 


17  June  2006  Data  (Hierarchy  Organization) 


17-Jun-06 

Identification  Attempts 

who 

what 

where 

when 

number 

Time  Player  Team 

Month 

Day 

Time 

correct 

3:06:34 

3:10:14 

3:11:55 

3:15:06 

3:16:43 

3:16:57 

3:18:30 

3:20:35 

3:21:43 

3:23:09 

3:23:26 

3:24:05 

3:24:13 

3:25:43 

3:25:47 

3:26:17 

3:26:54 

3:27:33 

3:29:03 

3:29:07 

3:30:23 

3:30:36 

3:30:48 

3:31:27 

3:32:05 

3:32:35 

3:34:29 

3:34:43 

3:34:45 

3:35:47 

3:36:15 

3:38:48 

3:39:30 

3:40:45 

3:41:46 

3:41:59 

3:47:51 

3:47:57 


Game  Started 


13 

13 

5 

15 


13 

13 

5 

13 


17 

16 

5 

17 

13 


13 


10 

10 

10 


5 

13 


When 

When 

What 

What 

When 

When 

What 

When 

When 


T 


Who 

When 

Who 

What 

Who 

When 

When 

When 

When 

When 

When 

When 

When 

What 

When 

When 

Who 

Where 

Who 

When 

Who 

Who 


Violet 

high  visability 

5th 

daylight 

1 

Purple 

Train  Station 

0 

violet 

1 

embassy 

0 

Purple 

Train  Station 

Tauland 

0 

Purple 

dignitary 

tauland 

0 

violet 

embassy 

1 

Dignitary 

Tauland 

5th 

11:00am 

2 

4  Where 

Azur 

13  When 

Purple 

Dignitary 

Tauland 

4  Where 

Tauland 

i  J  When 

violet 

financial  summit 

chiland 

13  When 

Violet 

dignitary 

Tauland 

5th 


5th 


5th 


13 


When 


violet 
violet 
The  Lion 
Violet 

violet 

purple _ 

Violet 

violet 

violet 

violet 

violet 

Violet 

violet 

violet 

Azur 

brown 

violet 

violet 

Violet 

violet 


11:00am 


11:00am 


11:00am 


embassy 

Psiland 

April 

5th 

4 

epsilonland  embassy 

tauland 

2 

Tauland  Embassy 

Epsilonland 

April 

10th 

1200am 

2 

dignitary 

Tauland 

June 

5th 

11:00am 

3 

11:00 

0 

Train  Station 

tauland 

June 

5th 

11:00am 

3 

Train  Station 

tauland 

June 

5th 

1 1 :00am 

2 

embassy 

Tauland 

June 

5th 

1 1 :00am 

3 

financial  summit 

chiland 

June 

5th 

11:00am 

3 

financial  summit 

chiland 

June 

5th 

11:00am 

3 

financial  summit 

psiland 

June 

5th 

11:00am 

4 

Epsilon  Dignitary 

Tauland 

1 

embassy 

Tauland 

June 

5th 

11:00am 

3 

psiland 

5th 

11:00am 

4 

financial  institution 

Chiland 

April 

5th 

4 

Embassy 

Epsilonland 

August 

22nd 

05:00am 

0 

dignitary 

Epsilonland 

April 

10th 

8:00pm 

1 

embassy 

tauland 

June 

5th 

11:00am 

3 

dignitary 

Psiland 

April 

5th 

12:00am 

4 

Train  Station 

Tauland 

April 

5th 

3 

embassy 

Tauland 

June 

15th 

11:00am 

2 

23 


3:48:01 

12 

What 

violet 

3:48:48 

10 

When 

violet 

3:48:53 

4 

Where 

3:48:55 

8 

When 

Violet 

3:49:07 

When 

3:49:16 

9 

Where 

Coral 

3:49:27 

2 

When 

Purple 

3:49:44 

14 

What 

Purple 

3:49:54 

8 

When 

Violet 

3:50:14 

10 

When 

violet 

3:50:28 

7 

Where 

Gold 

3:50:37 

17 

Who 

Violet 

3:50:38 

10 

When 

violet 

3:50:55 

10 

When 

violet 

3:51:04 

1 

Who 

Brown 

3:51:22 

5 

What 

Violet 

3:51:31 

2 

When 

Purple 

3:51:50 

15 

What 

3:52:43 

12 

What 

violet 

3:53:36  5  What  Violet 


3:53:57 


When 


Violet 
Violet  or 
Coral 

Violet 

Violet 

violet 

violet 


3:56:41 

10 

When 

violet 

3:57:15 

12 

What 

violet 

3:57:39 

16 

Who 

violet 

3:58:47 

10 

When 

violet 

3:58:52 

6 

When 

3:59:16 

4 

Where 

3:59:21 

10 

When 

violet 

3:59:25 

5 

What 

violet 

3:59:56 

11 

Where 

Azur 

4:00:09 

10 

When 

violet 

4:01:10 

1 

Who 

Azur 

4:01:52 

10 

When 

coral 

4:01:54 

1 

Who 

Azur 

4:02:21 

1 

Who 

Brown 

4:02:55 

10 

When 

violet 

4:03:26 

7 

Where 

Coral 

4:03:34 

1 

Who 

Brown 

4:04:58 

1 

Who 

Violet 

4:05:33 

10 

When 

violet 

4:08:40 

15 

What 

violet 

4:08:43 

2 

When 

Purple 

4:08:44 

15 

What 

violet 

embassy 

tauland 

June 

15th 

08:00am 

1 

embassy 

psiland 

June 

5th 

1 1 :00am 

4 

Tauland 

0 

Coalition  Embassy 

Psiland 

June 

5th 

11:00am 

4 

Psiland 

1 

Dignitary 

Psiland 

June 

1 

financial  institution 

psiland 

June 

5th 

11:00am 

4 

Coalition  Embassy 

Psiland 

June 

1st 

5:00pm 

1 

Coalition  Embassy 

Psiland 

June 

5th 

1 1 :00am 

4 

embassy 

epsilonland 

June 

5th 

11:00am 

3 

financial  institution 

Chiland 

anytime 

2 

Tauland  Embassy 

Epsilonland 

April 

5 

9:00pm 

3 

financial  summit 

Psiland 

June 

5th 

11:00am 

4 

financial  summit 

epsilonland 

July 

5th 

1 1 :00am 

3 

Train  Station 

Tauland 

April 

5th 

2 

embassy 

Tauland 

June 

18th 

1:00pm 

1 

financial  institution 

Psiland 

January 

5th 

11:00am 

4 

high  visability 

during  the 
day 

0 

Epsilon  Embassy 

Tauland 

june 

1 

8:00am 

1 

dignitary 

Epsilonland 

1 

Psiland 

April- 

December 

5 

11:00am 

4 

embassy 

psiland 

daytime 

1 

embassy 

epsilonland 

June 

day  time 

1 

dignitary 

Epsilonland 

April 

10 

12:00pm 

1 

conference 

epsilonland 

april 

10 

2:00 

2 

Epsilon  Embassy 

psiland 

2 

The  Lion 

tauland 

june 

5 

11:00am 

3 

embassy 

psiland 

june 

1 

8:00am 

2 

embassy 

Psiland 

april 

5th 

4 

psiland 

June 

5th 

11:00am 

4 

embassy 

Epsilonland's 

Embassy 

Tauland 

0 

0 

embassy 

psiland 

june 

5 

11:00am 

4 

embassy 

psiland 

June 

during  the 
day 

2 

Espilonland  Embassy 

Tauland 

August 

22 

3:00am 

0 

dignatary 

psiland 

june 

5 

1 1 :00am 

4 

Conference 

Epsilonland 

April 

10 

12:00pm 

1 

embassy 

psiland 

june 

5 

11:00am 

3 

embassy 

Epsilonland 

April 

10 

1:00pm 

1 

Embassy 

Epsilonland 

April 

10 

4:00pm 

1 

buildings 

psiland 

june 

5 

11:00am 

4 

Embassy 

Psiland 

work  day 

10:00am 

1 

Tauland  Embassy 

Epsilonland 

April 

10 

5:00pm 

1 

Tauland  Embassy 

Epsilonland 

April 

10 

4:00 

2 

dignatary 

psiland 

june 

5 

11:00am 

4 

1 

dignitary 

psiland 

June 

5 

11:00am 

3 

1 

24 


22  June  2006  Data  (Edge  Organization) 


22-Jun-06 

time  source 

who 

Identification  Attempts 

what  where 

Month 

when 

Day  Time 

number 

correct 

19:00:46 

Game  Started 

19:04:41 

13 

Violet 

Tauland  Embassy 

Epsilon 

April 

5 

11:00am 

4 

19:06:15 

4 

Violet  group 

1 

19:06:23 

13 

Violet  Group 

Financial  Institution 

Tauland 

April 

5 

11:00am 

5 

19:07:37 

4 

financial  institution 

1 

19:07:50 

4 

violet  group 

financial  instituion 

2 

19:07:50 

8 

the  violet  group 

financial  institution 

2 

19:08:13 

16 

coral 

0 

19:09:01 

16 

Epsilonland 

0 

19:09:16 

16 

International  Conference 

0 

19:09:27 

4 

Violet  group 

financial  institution 

April 

5 

4 

19:11:24 

12 

the  coral  group 

embassy 

Epsilonland 

April 

10 

09:00am 

1 

19:11:38 

8 

April 

5 

2 

omega- 

19:12:32 

13 

violet  group 

financial  institution 

lands 

April 

5 

11:00am 

5 

19:13:26 

16 

violet 

1 

19:13:51 

6 

Violet  group 

embassy 

Tauland 

April 

10 

11:00am 

3 

19:13:58 

16 

April 

10 

11:00am 

2 

19:14:17 

12 

coral 

embassy 

epsilonland 

April 

10 

11:00am 

2 

19:14:43 

14 

Lion 

Financial  Institution 

Psiland 

April 

5 

11:00am 

5 

19:14:45 

6 

Violet 

Financial 

Psiland 

April 

10 

11:00am 

4 

19:14:46 

12 

coral 

embassy 

tauland 

April 

10 

11:00am 

2 

19:15:01 

8 

11:00am 

0 

19:15:16 

12 

violet 

embassy 

epsilonland 

April 

5 

11:00am 

4 

19:15:38 

8 

April 

5 

23:00 

2 

19:15:50 

4 

Violet  group 

embassy 

Psiland 

June 

10 

11:00am 

3 

19:15:53 

11 

April 

10 

11:00am 

2 

19:16:00 

12 

coral 

embassy 

epsilonland 

April 

10 

11:00am 

2 

19:16:07 

13 

Violet  Group 

Financial  Institution 

Psiland 

April 

5 

11:00am 

6 

19:16:18 

12 

coral 

embassy 

tauland 

April 

10 

100:00:00 

1 

19:16:29 

8 

April 

10 

1 

19:16:33 

14 

Lion 

Financial  Institution 

Psiland 

April 

5 

11:00am 

5 

19:16:35 

13 

Violet  Group 

Financial  Instituation 

Tauland 

April 

5 

11:00am 

5 

19:16:37 

12 

coral 

embassy 

tauland 

April 

10 

11:00pm 

1 

19:16:41 

15 

violet 

financial  institution 

omegaland 

April 

5 

11:00am 

5 

19:17:08 

5 

Violet 

Financial  Institution 

April 

5 

11:00am 

5 

19:17:11 

13 

Violet  Group 

Financial  Instituation 

Upsilonland 

April 

5 

11:00am 

5 

19:17:13 

15 

violet 

financial  institution 

psiland 

April 

5 

11:00am 

6 

19:17:38 

13 

Violet  Group 

Financial  Institution 

Chiland 

April 

5 

11:00am 

5 

19:17:40 

15 

violet 

financial  institution 

tauland 

April 

5 

11:00am 

5 

19:17:47 

2 

April 

5 

11:00am 

3 

19:18:18 

8 

tauland 

0 

19:18:31 

14 

Lion 

Financial  Institution 

Psiland 

April 

10 

11:00am 

4 

19:18:46 

4 

violet  group 

financial  instituion 

chiland 

June 

5 

11:00am 

4 

25 


19:18:48 

3 

Violet 

financial  institution 

Psiland 

April 

5 

11:00am 

19:18:52 

8 

the  violet  group 

financial  insititution 

epsilonland 

April 

10 

11:00am 

19:18:58 

15 

coral 

financial  institution 

tauland 

April 

10 

11:00am 

19:19:01 

13 

Violet  Group 

embassy 

Epsilonland 

April 

5 

11:00am 

19:19:07 

9 

the  violet  group 

Tauland  embassy 

Epsilonland 

April 

10 

11:00am 

19:19:16 

7 

Violet  group 

financial  institution 

Psiland 

April 

5 

11:00am 

19:19:26 

12 

violet 

embassy 

epsilonland 

April 

5 

11:00am 

19:19:48 

12 

violet 

embassy 

tauland 

April 

5 

11:00am 

19:20:02 

15 

coral 

financial  institution 

psiland 

April 

10 

11:00am 

19:20:11 

8 

omegaland 

19:20:12 

14 

Lion 

Financial  Institution 

Psiland 

April 

10 

11:00pm 

19:20:14 

4 

violet  group 

embassy 

psiland 

April 

5 

11:00am 

19:20:16 

5 

Violet 

Financial  Institution 

Psiland 

April 

5 

11:00am 

19:20:22 

13 

Violet  Group 

Tauland's  embassy 

Epsilonland 

April 

5 

11:00am 

19:20:57 

6 

Violet 

financial 

Tauland 

April 

5 

11:00am 

19:21:00 

2 

Violet 

April 

5 

11:00am 

19:21:56 

4 

violet  group 

financial  institution 

Psiland 

April 

5 

11:00am 

19:22:07 

12 

violet 

embassy 

epsilonland 

April 

5 

11:00am 

19:23:17 

12 

brown 

embassy 

epsilonland 

April 

10 

11:00am 

19:23:39 

12 

brown 

embassy 

epsilonland 

April 

10 

11:00pm 

19:24:36 

4 

Violet  group 

financial  institutions 

Chiland 

April 

5 

11:00am 

19:25:21 

12 

brown 

embassy 

tauland 

April 

10 

11:00am 

19:25:28 

11 

Lion  and  Violet 

Epsilonland 

April 

5 

11:00am 

19:25:41 

12 

brown 

embassy 

tauland 

April 

5 

11:00am 

19:26:05 

13 

Violet  Group 

Financial  Institution 

Omegaland 

April 

5 

11:00am 

19:27:21 

8 

11:00am 

19:27:23 

4 

violet  group 

financial  institution 

Chiland 

April 

5 

11:00am 

19:28:52 

3 

Violet 

Financial  Institution 

Tauland 

April 

5 

11:00am 

19:29:00 

15 

violet 

financial  institution 

epsilonland 

April 

5 

11:00am 

19:29:10 

12 

violet 

dignitary 

epsilonland 

April 

5 

11:00am 

19:29:25 

3 

Violet  group 

financial  institution 

Omegaland 

April 

5 

11:00am 

19:29:42 

12 

violet 

dignitary 

epsilonland 

April 

10 

11:00am 

19:30:04 

8 

violet 

financial 

omegaland 

April 

5 

11:00am 

19:30:26 

12 

brown 

dignitary 

epsilonland 

April 

10 

11:00am 

19:30:43 

9 

The  Lion 

Tauland  Embassy 

Epsilonland 

April 

10 

11:00am 

19:30:47 

15 

violet 

financial  institution 

omegaland 

April 

5 

11:00am 

19:31:09 

15 

violet 

dignitary 

omegaland 

April 

5 

11:00am 

19:31:19 

3 

Violet  group 

financial  institution 

Epsilonland 

April 

5 

11:00am 

19:31:55 

12 

violet 

dignitary 

tauland 

April 

5 

11:00am 

19:32:13 

6 

Violet 

Financial 

Psiland 

April 

5 

11:00am 

19:32:17 

2 

Violet  Group 

Financial  Institution 

Psiland 

April 

5 

11:00am 

19:32:19 

12 

violet 

dignitary 

epsilonland 

April 

5 

11:00am 

19:32:20 

8 

dignitary 

19:32:25 

15 

violet 

dignitary 

epsilonland 

April 

5 

11:00am 

19:32:46 

15 

violet 

dignitary 

tauland 

April 

5 

11:00am 

19:33:23 

14 

Lion 

Financial  Institution 

Chiland 

April 

10 

11:00am 

19:33:45 

15 

brown 

dignitary 

tauland 

April 

10 

11:00am 

19:34:12 

14 

Lion 

Financial  Institution 

Psiland 

April 

10 

11:00am 

19:36:12 

14 

Lion 

Embassy 

Tauland 

April 

10 

11:00am 

19:36:46 

15 

violet 

dignitary 

tauland 

April 

5 

11:00am 

6 

4 

3 

4 

3 

6 

4 

4 

4 

0 

3 

5 

6 

4 

4 

4 

6 

4 

2 

1 

5 

2 

3 

3 

5 

1 

5 

5 

5 

4 

5 

3 

4 

2 

2 

5 

4 

5 

4 

5 

6 

4 
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4 
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2 

4 

2 
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19:37:56 

11 

Violet  and  the  Lion 

financial  institution 

tauland 

April 

5 

11:00am 

4 

19:38:02 

15 

violet 

financial  institution 

tauland 

April 

11 

11:00am 

4 

19:38:54 

10 

Violet  Group 

April 

5 

11:00am 

4 

19:39:50 

5 

Violet 

Financial  Institution 

Omegaland 

April 

5 

11:00 

4 

19:40:04 

12 

brown 

embassy 

epsilonland 

April 

10 

11:00am 

2 

19:40:05 

4 

violet  group 

financial  institution 

Psiland 

May 

5 

11:00am 

5 

19:40:45 

14 

The  Lion  Group 

Financial  Institutions 

Chiland 

April 

10 

11:00am 

3 

19:41:13 

14 

The  Lion  Group 

Financial  Institution 

Psiland 

April 

10 

11:00am 

4 

19:41:42 

1 

Violet  group 

the  Tauland  embassy 

Epsilonland 

June 

10 

11:00am 

2 

19:41:43 

8 

violet 

dignitary 

psiland 

April 

10 

11:00am 

4 

19:41:52 

14 

The  Lion  Group 

Embassy 

Tauland 

April 

10 

11:00am 

2 

19:42:48 

15 

violet 

financial  institutions 

omegaland 

April 

5 

11:00am 

5 

19:43:10 

15 

violet 

financial  institution 

tauland 

April 

5 

11:00am 

5 

19:43:51 

15 

purple 

financial  institution 

tauland 

April 

10 

11:00am 

3 

19:43:57 

12 

gold 

embassy 

epsilonland 

April 

10 

11:00am 

2 

19:44:30 

10 

financial  institution 

1 

19:45:50 

15 

violet 

financial  institution 

tauland 

April 

5 

11:00am 

5 

19:46:30 

15 

violet 

financial  institutions 

omegaland 

April 

5 

11:00am 

5 

19:49:08 

15 

violet 

financial  institution 

psiland 

April 

5 

11:00am 

6 

19:49:28 

15 

violet 

dignitary 

tauland 

April 

5 

11:00am 

4 

19:50:07 

11 

Violet  and  lion 

financial  institution 

omegaland 

April 

5 

11:00am 

4 

19:50:37 

15 

violet 

financial  insitution 

epsilonland 

April 

5 

11:00am 

5 

19:50:53 

10 

Violet  Group 

Financial  Institution 

Tauland 

April 

5 

11:00am 

5 

19:50:56 

15 

violet 

dignitary 

epsilonland 

April 

5 

11:00am 

4 

19:54:29 

5 

Lion  with  Violet 

Financial 

Omegaland 

April 

5 

11:00 

2 

19:58:58 

14 

The  Lion  Group 

Financial  Institution 

Chiland 

April 

25 

11:00am 

3 

19:59:41 

14 

The  Lion  Group 

Financial  Institution 

Psiland 

April 

25 

11:00am 

4 

The  Correct 

Answer 

Violet 

Financial  Institution 

Psiland 

April 

5th 

11:00am 

23  June  2006  Data  (Hierarchy  Organization) 


23-Jun-06 

Time  Player 

Team 

Identification  Attempts 

who 

what 

where 

Month 

when 

Day 

Time 

number 

correct 

18:46:51 

Game  Started 

18:53:52 

3 

Who 

Coral  Group 

Financial  Institution 

Chiland 

1 

18:54:07 

13 

What 

coalition  member  embassy 

0 

18:55:00 

3 

Who 

Coral 

Embassy 

Chiland 

0 

18:56:08 

15 

What 

azur 

embassy 

psiland 

1 

18:56:21 

What 

Azur 

Embassy 

0 

18:56:48 

What 

azur  group 

coalition  member  embassy 

0 

18:57:23 

5 

Who 

Coral  group 

embassy 

Tauland 

0 

18:57:44 

15 

What 

azur 

embassy 

chiland 

0 

18:59:58 

9 

When 

the  Violet  group 

1 

19:00:18 

7 

What 

azur 

embassy 

epsilonland  daytime 

0 

19:00:51 

6 

When 

June 

5 

morning 

1 

19:01:08 

6 

When 

a  visiting  dignitary 

0 

19:02:40 

6 

When 

June 

5 

11:00am 

2 

27 


19:02:53 

19:03:40 

19:03:52 

14 

Where 

Violet  group 

Embassy 

Psiland 

May 

10 

11:00am 

6 

When 

Epsilonland 

11 

Violet  group 

embassy 

Psiland 

June 

5 

11:00am 

19:05:05 

5 

Who 

Coral 

Embassy 

Tauland 

December 

19 

10:00pm 

19:05:26 

4 

Where 

Azur 

Embassy 

Epsilonland 

June 

15 

12:00pm 

19:07:10 

3 

Who 

Azur 

Embassy 

Epsilonland 

19:07:52 

9 

When 

violet  group 

embassy 

Chiland 

April 

5 

11:00am 

19:07:59 

10 

Where 

Violet  group 

Financial  Institution 

19:08:15 

15 

What 

azur 

embassy 

tauland 

june 

19:08:53 

15 

What 

azur 

embassy 

psiland 

june 

19:10:07 

What 

Coral  Group 

Embassy 

Psiland 

June 

15 

11:00am 

19:10:28 

15 

What 

azur 

embassy 

epsilonland 

june 

19:10:28 

16 

When 

Violet 

Coalition  Member  Embassy 

Psiland 

June 

5 

11:00am 

19:10:41 

6 

When 

Purple  group 

dignitaries 

Epsilonland 

June 

5 

11:00am 

19:10:48 

14 

Where 

Coral 

Dignitary 

Psiland 

19:10:49 

9 

When 

Violet  group 

visiting  dignitary 

Psiland 

April 

5  11:00am 

19:11:14 

Ml 

Where 

Violet 

Embassy 

psiland 

10 

19:11:29 

3 

Who 

Coral 

Financial  Institution 

Psiland 

19:11:53 

n 

Where 

Coral 

embassy 

Psiland 

June 

10 

19:12:12 

B 

Where 

Violet 

Dignitary 

Psiland 

June 

10 

19:12:56  9  When  violet  group  embassy  psiland  April  5  11:00am 


19:14:50 

nn 

Who 

violet 

19:15:34 

5 

Who 

Coral  group 

Coalition  member  embassy 

Tauland 

December 

19 

12:00am 

19:16:09 

4 

Where 

Coral 

dignitary 

epsilonland 

June 

5 

5:00pm 

19:17:34 

10 

Where 

Tauland 

19:19:32 

6 

When 

Chartreuse  group 

dignitaries 

epsilonland 

June 

5 

11:00pm 

19:20:33 

4 

Where 

Purple 

19:20:38 

4 

Where 

Gold 

19:20:40 

1 

What 

Chartreuse  group 

coalition  embassy 

Epsilonland 

June 

15 

3:00pm 

19:20:44 

4 

Where 

Brown 

19:21:04 

2 

When 

June 

5 

11:00am 

19:21:15 

15 

What 

azur 

embassy 

psiland 

june 

19:22:38 

8 

Who 

violet 

embassy 

chiland 

19:22:42 

4 

Where 

Chiland 

19:22:50 

4 

Where 

Psiland 

19:22:57 

4 

Where 

Omegaland 

19:23:09 

12 

Where 

Coral 

Financial  Institution 

Psiland 

June 

10 

19:23:38 

1 

What 

charteuse  group 

coalition  embassy 

epsilonland 

June 

15 

1:00pm 

19:24:31 

15 

What 

violet 

embassy 

psiland 

june 

19:25:03 

7 

What 

azur  group 

embassy 

omegaland 

june 

19:27:44 

1 

What 

The  violet  group 

coalition  member  embassy 

epsilonland 

June 

15 

1:00pm 

19:27:47 

7 

What 

azur 

embassy 

omegaland 

June 

1 :00pm 

19:28:02 

14 

Where 

Violet 

Dignitary 

Psiland 

June 

10 

3:00pm 

19:29:11 

6 

When 

purple  group 

dignitaries,  embassies 

epsilonland 

June 

5 

11:00am 

19:32:46 

3 

Who 

Coral  Group 

Epsilonland 

19:33:04 

Wl 

Who 

violet 

embassy 

chiland 

June 

3 

1:00am 

19:33:37 

9 

When 

the  Violet  group 

visiting  dignitary 

Psiland 

April 

5 

11:00am 

19:33:47 

What 

June 

1 

1 :00pm 

19:34:11 

12 

Where 

Azur  group 

Tauland  Embassy 

Epsilonland 

June 

10 

19:34:54 

15 

What 

azur 

embassy 

psiland 

june 

3 

0 

4 

0 

0 

0 

4 

2 

0 

1 

2 

0 

4 

2 

1 

5 

2 

2 

1 

2 

5 

1 

0 

1 

0 

1 

0 

0 

0 

0 

2 

1 

1 

0 

1 

0 

2 

0 

2 

0 

1 

0 

2 

2 

0 

1 

5 

0 

0 

1 


28 


19:35:14 

15 

What 

violet 

embassy 

psiland 

june 

19:35:45 

6 

When 

the  jackal 

embassies 

epsilonland 

June 

5 

11:00am 

19:35:46 

What 

the  Chartreuse 
group 

coalition  embassy 

Psiland 

June 

1 

1:00pm 

19:36:53 

3 

Who 

The  Coral  Group 

Embassy 

Epsilonland 

June 

13 

2:00am 

19:37:11 

9 

When 

violet  group 

embassy 

chiland 

June 

5 

11:00am 

19:38:04 

5 

Who 

Coral  group 

coalition  member  embassy 

Epsilonland 

January 

24 

4:00am 

19:38:49 

6 

When 

The  Jackal 

Embassies 

Tauland 

June 

5 

11:00am 

19:39:40 

3 

Who 

Violet  Group 

19:41:08 

isfl 

What 

Coral  Group 

Coalition  Member  Embassy 

Pisland 

October 

3 

2:00pm 

19:41:29 

4 

Where 

the  Lion 

attack  dignitaries 

Psiland 

June 

15 

11:00am 

19:42:46 

3 

Who 

Violet  Group 

embassy 

Psiland 

19:43:28 

13 

What 

Coral 

Coalition  Member's 
Embassy 

Pisland 

December 

15 

3:00am 

19:45:54 

3 

Who 

The  Violet  Group 

Embassy 

Psiland 

June 

21 

1 :00pm 

^■1 

Correct  Answer 

Violet 

Financial  Institution 

Psiland 

April 

5th 

11:00am 

Appendix  B  -  ELICIT  game  suggestions 

A  few  modest  extensions  to  the  ELICIT  exercise  would  provide  for  improved  realism  and  richer 
experimental  output.  With  these  extensions,  ELICIT  would  more  faithfully  represent  the  effect  of 
changes  in  organization  structure  on  team  performance  for  this  C2  task;  and  ELICIT  could  be 
modeled  in  POW-ER  with  greater  predictive  capability  and  should  provide  further  qualitative  and 
quantitative  distinction  between  Hierarchy  and  Edge  organizations.  We  offer  the  following  list  of 
suggested  changes  to  the  experiment  protocol  and  software  for  consideration  by  ELICIT’s  developers 
and  its  users,  in  the  event  that  an  ELICIT  users'  group  is  formed. 

1 .  Give  players  some  “factoid  source  reliability  hints”  to  help  them  judge  the  “source 
reliability”  (qualitatively  or  as  a  percentage)  for  each  factoid.  Players  could,  for  example, 
be  given  a  characterization  of  the  source  for  each  factoid  as:  “reliable”,  “unknown”  or 
“potential  source  of  misinformation”.  This  source  information  could  be  given  to  different 
players  than  the  ones  who  received  the  original  factoid,  so  players  would  have  to 
exchange  information  to  rate  the  reliability  of  the  source  for  each  factoid. 

2.  Allow  the  players  to  request  specific  information  from  the  coordinator  (in  Hierarchy  mode) 
and/or  other  players  (in  both  modes,  but  especially  in  Edge  mode)  such  as:  “Where  is  the 
Coral  group?”  This  would  address  our  concern  about  introducing  alternative  forms  of 
exception  handling  to  ELICIT  whose  availability  and  effectiveness  would  differ  with 
different  organization  structures. 

3.  Penalize  players  for  wrong  answers,  perhaps  in  terms  of  reputation  points  and  give  them 
feedback  about  wrong  answers.  For  instance,  each  player  could  begin  with  ten  points 
and  lose  one  for  every  wrong  answer,  or  gain  one  point  for  every  correct  answer.  Each 
player  could  be  given  a  secondary  goal  of  maximizing  their  points.  The  game 
administrator  might  be  the  objective  observer  who  could  manage  these  points  and 
communicate  them  to  players.  This  would  address  our  concern  about  providing  players 
with  feedback. 

4.  The  game  might  have  some  rules  for  early  termination  when  a  single  player,  a  plurality  of 
players  or  a  majority  of  players  achieves  the  correct  answer  for  the  game.  This  would 
allow  researchers  to  compare  empirical  vs.  predicted  (by  the  simulation)  project 
completion  durations  which  we  cannot  do  now.  This  also  seems  to  us  to  increase  realism 
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in  the  exercise.  If  a  potential  adversary  were  identified  in  a  real  counterespionage  or 
counterterrorism  scenario,  presumably  they  might  be  able  to  be  apprehended  for 
questioning  and  their  person  and  premises  searched,  etc.  which  could  begin  to  confirm  or 
disconfirm  the  diagnosis. 
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CRGP  Collaborator/  for  Research  on  Global  Projects 


Motivation 


■  Edge  Organization  definition 

►  No  headquarters  to  rely  upon 

►  Requires:  shared  awareness  /  self  synchronization 

■  Knowledge  flow  is  especially  critical  for  Edge  Organizations 

►  High  levels  of  strategic  &  operational  knowledge  needed  at  nodes 

►  Enables  “agility”  in  an  uncertain  environment 

►  Understanding  knowledge  growth  &  decay  in  Edge  organizations  - 
critical  for  optimizing  performance 
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Research  Questions 


■  How  can  we  model  and  simulate  Edge  vs.  Hierarchy 
organizational  forms  engaged  in  similar  project-oriented 
tasks,  taking  account  of  the  impacts  of  individual 
learning  and  forgetting  on  performance  outcomes  for 
the  two  structures? 

►  How  can  individual  skill  acquisition  and  decay  be 
computationally  modeled,  calibrated,  and  validated? 

►  How  are  Edge  vs.  Hierarchy  organizations  and  projects 
effected  by  the  sum  of  individual  participants’  skill  growth  and 
decay? 
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Points  of  Departure 


■  Hypothesis  Testing  of  Edge  Organizations 

•  Nissen,  10th  ICCRTS,  2005 

•  Orrand  Nissen,  11th  ICCRTS,  2006 

■  Cognitive  Science 

►  Learning  and  Forgetting  rates 

•  Anderson,  2005 

•  Sikstrom  and  Jaber,  2004  and  2002 

►  Skill  Classification 

•  Dar-EI  et  al,  1995 
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CRGP  Collaborator/  for  Research  on  Global  Projects 


Theoretical  Point  of  Departure 

Skill  Context  (Dar-EI  et  al.,  1995) 


■  Different  skill  types  seem  to  have  different  learning  curves 
►  Ranging  from  highly  cognitive  to  highly  motor  skills 

Modeling  High  Cog  to  High  Motor 


♦  Hi  Cog 
■  More  Cog 
More  Motor 
Hi  Motor 
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Conceptual  Model 

Individual  Skill  Acquisition  and  Decay 

■  Extensions  to  POW-ER  computational  modeling 

■  Develop  fine-grained  agent  knowledge  metric 

►  Provide  for  dynamic,  continuous  knowledge  overtime 

■  Focus  on  individual  knowledge 

►  Inflows  through  OJT 

►  Outflows  through  decay 


High 

Medium 

Low 

Unskilled 
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CRGP  Collaborator/  for  Research  on  Global  Projects 

AROUSAL  Model 


POW-ER  3.2 


l  POWer  Editor 

*C:/Documents 

and  Settings! 

fdjmackin/Desktop/Orig  Arousal 

□0® 

jj 


Arousal 


CRGP  Collaborator/  for  Research  on  Global  Projects 


AROUSAL  Model 


POW-ER  3.2 


I  Settings/djmackin/Desktop/Arousal_v8. vpx 


Ji3Q 


jJ 

Static  |  Learning  | 


Time  (normalized) 


Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 


Hi  Cog 
More  Cog 
More  Motor 
Hi  Motor 
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Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 


Hi  Cog 
More  Cog 
More  Motor 
Hi  Motor 
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Time  (normalized) 


Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 
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More  Cog 
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Time  (normalized) 


Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 
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More  Cog 
More  Motor 
Hi  Motor 
Group  Time 
Individual  Time 


ICCRTS  Conference  —  Newport,  Rl,  June  19-21,  2007  —  Organizational  Issues  Track  (1-092)  13 


Time  (normalized) 


Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 


Hi  Cog 

More  Cog 

More  Motor 

Hi  Motor 

Group  Time 

Individual  Time 

POW-ER  3.2  Individual  Time 
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Time  (normalized) 


Collaborator/  for  Research  on  Global  Projects 


Empirical  Validation  of  Learning  Rates 

Arousal  Exercise 


Dar-EI  Learning  Curves  Plotted  Against 
Observed  Individual  &  Group  Learning  Rates 


1.4 

1.2 

1.0 

0.8 

0.6 

0.4 

0.2 

0.0 


Hi  Cog 

More  Cog 

More  Motor 

Hi  Motor 

Group  Time 

Individual  Time 

POW-ER  3.2  Individual  Time 

POW-ER  3.2  Group  Time 
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Individual  Duration  (minutes) 


Collaborator/  for  Research  on  Global  Projects 


POW-ER  Validation 

Empirical  vs.  Predicted  Individual  Performance 


n  AROUSAL 
□  POW-ER  3.2 


AROUSAL  Quarter 
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Group  Duration  (minutes) 


Collaborator/  for  Research  on  Global  Projects 


POW-ER  Validation 

Empirical  vs.  Predicted  Group  Performance 


Q2  Q3  Q4  Q5  Q6  Q7  Q8 


■  AROUSAL 
□  POW-ER  3.2 


AROUSAL  Quarter 
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Organizational  Level  POW-ER  Models 


Empirical  findings  from  AROUSAL  learning  and  forgetting 


Metric 

Individual  Data 

Group  Data 

Empirical 

POW-ER  3.2 

Empirical 

POW-ER  3.2 

Summed  individual 
durations  (based  on 
initial  period,  without 
subsequent  learning) 

161  minutes 

161  minutes 

406  minutes 

406  minutes 

Duration  (with  learning) 

106  minutes 

103  minutes 

235  minutes 

233  minutes 

Percent  Savings  from 
Learning 

34.2% 

36.0% 

42.1% 

42.6% 

ICCRTS  Conference  —  Newport,  Rl,  June  19-21,  2007  —  Organizational  Issues  Track  (1-092)  18 


Collaborator/  for  Research  on  Global  Projects 


ELICIT  Exercise 


■  Anti-terrorist,  intelligence  (knowledge)  sharing  game 

■  17  players 

■  Either  Edge  or  Hierarchy  organizations 

►  All  players  may  share  information  with  each  other 

►  Hierarchy  is  limited  to  viewing  own  team’s  website 

►  No  talking 

■  Each  player  required  to  identify  target 

►  Who,  what,  when,  where 

■  Allowed  approximately  60  minutes 
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POW-ER  Model  Validation 


ELICIT  Exercise 


Hierarchy 
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Edge 
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ELICIT  Exercise 


Correct  Responses  in  10-minute  Intervals 

For  Different  Organizational  Forms 


</) 

i 


<D 

s_ 

O 

o 


□  Hierarchy  (2  game  avg) 
■  Edge 


10  20  30  40  50  60 


10-Minute  Intervals  (non-cumulative) 
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POW-ER  Experimental  Results 

3  Exercise  Rounds:  3  day  delay  after  2nd  round 


Metric 

Hierarchy  (3  Rounds) 

Mean  (Std.  deviation) 

Edge  (3  Rounds) 

Mean  (Std.  deviation) 

totals 

No  learning 

With  learning 
and  forgetting 

No  learning 

With  learning 
and  forgetting  i 

Duration  (minutes)* 

Coordination  (minutes)* 

Rework  (minutes) 

Functional  Exception 
Work  (minutes) 

Total  Work  (minutes)* 

Functional  Risk 

Process  Quality  Risk 

Cost  ($K)* 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


Empirical  Hier 


Trial 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


Empirical  Hier 
POW-ER  Hier 


Trial 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


Empirical  Edge 


Trial 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


Empirical  Edge 
POW-ER  Edge 


Trial 


Time  (minutes) 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


—  Empirical  Hier 

—  Empirical  Edge 


Trial 


Time  (minutes) 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 


—  Empirical  Hier 

—  Empirical  Edge 
POW-ER  Hier 


Trial 
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POW-ER  External  Validity 

Using  ELICIT  Observations  (Leweiing  &  Nissen,  2007) 
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—  Empirical  Hier 

- Empirical  Edge 

POW-ER  Hier 
POW-ER  Edge 


Trial 
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C2  Application 


■  Example:  Crew  training  (deployment  preparation) 

►  Consider  improvement  in  command’s  performance  through 
adoption  of  edge-like  organizational  qualities 

■  Leverage  experimental  results  to  develop  and  test 
new  command  models 

►  To  predict  project  lengths  for  a  single  project 

►  To  consider  impacts  of  other  agent-based  knowledge 
interventions 

•  e.g.,  training  and  mentoring 
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Theoretical  Contributions 


■  Produced  quantitative  analysis  of  how  micro¬ 
behaviors  (learning  and  forgetting)  affect 
organizational  performance,  extending  our 
understanding  of  organizational  learning 

■  Calibrated  and  validated  tool  to  develop  and  test 
individual  knowledge  flow  impacts  on  Edge  and 
other  organizational  forms 
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Next  Steps 


■  Develop  and  validate  further  via  future  ELICIT 
experiments,  so  that  we  can 

Improve  our  predictions  of  project  lengths  for  a  single 
project 

►  Model  the  effects  of  other  knowledge  interventions 

•  Training,  mentoring 

•  Obsolescence,  interference 
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